# projects

name: Social media privacy project
created: 2024-06-21 14:53:48.297396
description:
  Re-analysis of interviews from Mannheimer, Sara. 2023. "Interviews regarding data curation for qualitative data reuse and big social research". Qualitative Data Repository. https://doi.org/10.5064/F6GWMU4O. QDR Main Collection.


# documents

name: BSR_interview_guide
description: Interview guide for Big Social Research interviews
filename: Mannheimer_BSR_Inter.pdf
created: 2024-06-21 18:14:58.622705
text_direction: LEFT_TO_RIGHT
contents:
  <p>Big Social Researchers Interview Guide Project Overview</p>
  <p>Project Title</p>
  <p>Connecting communities of practice: Using strategies from qualitative data curation to support big social research</p>
  <p> </p>
  <p>Research summary</p>
  <p>Big social data (such as social media and blogs) and archived qualitative data (such as interview transcripts, field notebooks, and diaries) are similar, but their respective communities of practice are under-connected. Research with both types of data repurpose existing social data to advance discoveries in social science. However, despite these similarities, big social research has not yet been widely framed as a form of qualitative data reuse, and qualitative data reuse has only begun to be discussed through a big social data lens. Qualitative data reuse is a more established practice, and therefore has more developed data curation strategies to support data sharing. My research investigates how data curation practices from each of these communities can inform the other for mutual benefit.  The research wil  use interviews of qualitative researchers, big social data researchers, and data curators to gain insights into different community approaches to research and data sharing.</p>
  <p> </p>
  <p>Research background</p>
  <p>This research asks: how can data curators best handle qualitative and big social data to support ethical, epistemological, and legal data sharing practices?</p>
  <p> </p>
  <p>My review of the literature revealed that there are six key issues that pose chal enges for both groups. During the interview, I wil  ask questions about your personal experience in each of these six topic areas, plus introductory and wrap up questions. The interview wil  take 60-75</p>
  <p>minutes.</p>
  <p> </p>
  <p>Thank you for taking the time to speak with me! Your interview wil  help to improve data curation practices across disciplines.</p>
  <p> </p>
  <p>1</p>
  <p>Interview questions</p>
  <p>We’l  start recording the interview now.</p>
  <p> </p>
  <p>Introductory question</p>
  <p>Tel  me about the type of research you do and what kind of data you produce.</p>
  <p> </p>
  <p>Identifying a specific example</p>
  <p>Please describe a recent time when you:</p>
  <p>-  collected big social data for research; or</p>
  <p>-  reused big social data that was col ected and shared by someone else; or</p>
  <p>-  prepared big social data for publication or sharing.</p>
  <p> </p>
  <p>Please also describe your data collection method (API, scraping, shared dataset, etc.) Was this example part of a grant-funded project that required specific treatment of the data?</p>
  <p> </p>
  <p>Did you have a data management plan?</p>
  <p> </p>
  <p>Did you publish any of the data from your example?</p>
  <p>-  Is the data published in a repository? Which one?</p>
  <p>-  What are your plans for storing, retaining, and deleting data in the future?</p>
  <p>-  Who has access to the data?</p>
  <p> </p>
  <p>Context</p>
  <p>Halavais (2015) suggests that “when we collect data from [social media] platforms (just as when we col ected data in traditional spaces), context matters.” However, the context of a social media post may be absent or difficult to understand. Social media posts are by nature short pieces of text, images, videos, etc, taken from a larger context of personal and public life. This out-of-context effect is only compounded when data are amassed at a large scale.</p>
  <p> </p>
  <p>1. Tel  me about a time (if any) during the process of your example when you considered the issue of maintaining and understanding the data’s context (i.e. contextual information about the community where the data was col ected, contextual information about respondents)?</p>
  <p>2</p>
  <p>-  What were your considerations? What concerns did you have? What factors helped you better understand/communicate context? What factors prevented you from understanding/communicating context? Was your research affected by incomplete contextual information?</p>
  <p>-  Did you consult with anyone, consider other research projects, or refer to literature, policies, or guidelines regarding this issue? Please explain.</p>
  <p>-  What strategies did you use to discern/understand context during your example?</p>
  <p>-  If the example includes publishing your own big social data, what strategies did you use to communicate context to future users?</p>
  <p> </p>
  <p>Data quality</p>
  <p>2. Tel  me about a time (if any) during the process of your example when you considered the issue of data quality (for example, missing data, bots, bias, quality of method)?</p>
  <p>-  How did you assess quality? What data quality issues arose?</p>
  <p>-  What data quality concerns did you have?</p>
  <p>-  What factors helped you better consider data quality issues? What factors prevented you from considering data quality?</p>
  <p>-  Did you consult with anyone, consider other research projects, or refer to literature, policies, or guidelines regarding this issue? Please explain.</p>
  <p>-  What strategies did you use (or did you see used) to communicate, describe, or clarify data quality issues in your example? Please describe in detail how those strategies helped you.</p>
  <p> </p>
  <p>Data comparability</p>
  <p>3. During your example, did you compare and/or combine multiple big social datasets?</p>
  <p>-  or: Did you consider comparability or interoperability of your dataset?</p>
  <p>-  Did you consult with anyone, consider other research projects, or refer to literature, policies, or guidelines regarding this issue? Please explain.</p>
  <p> </p>
  <p>If no:</p>
  <p>-  why not?</p>
  <p> </p>
  <p>If yes:</p>
  <p>-  Why (for what purpose) did you combine the datasets? How did this advance your research?</p>
  <p>3</p>
  <p>-  what dataset did you combine it with - where did that data come from? Your own or someone else’s?</p>
  <p>-  What strategies did you use to combine multiple qualitative datasets?</p>
  <p>-  what chal enges did you encounter and how did you address them?</p>
  <p> </p>
  <p>Informed consent</p>
  <p>4. Tel  me about a time (if any) during the process of your example when you considered informed consent.</p>
  <p>-  Did you consult with anyone, consider other research projects, or refer to literature, policies, or guidelines regarding this issue (including IRB)? Please explain.</p>
  <p>-  Have you used any other type of consent besides informed consent per se (e.g.</p>
  <p>broad consent, focus groups, community advisory boards)</p>
  <p>-  Did you feel that participants in your research would expect to give informed consent for the research?</p>
  <p> </p>
  <p>Privacy and confidentiality</p>
  <p>5. Tel  me about a time (if any) during the process of your example when you considered issues of privacy (e.g. protecting data during research, considering restricted access or TweetIDs only if publishing).</p>
  <p>-  Did you consult with anyone, consider other research projects, or refer to literature, policies, or guidelines regarding this issue? Please explain.</p>
  <p>-  During your example, what do you think the participants' expectations of privacy were?</p>
  <p>-  Did you feel that you had to make any compromises about participant privacy in order to conduct your research?</p>
  <p>-  What strategies did you use to address the issue of privacy?</p>
  <p> </p>
  <p>Intellectual property</p>
  <p>6. Tel  me about a time (if any) during the process of your example when you considered intel ectual property concerns when you conducted your research and/or published your data (e.g. social media platform terms of service, participant intel ectual property).</p>
  <p>-  Did you consult with anyone, consider other research projects, or refer to literature, policies, or guidelines regarding this issue? Please explain.</p>
  <p>-  Do you consider your research to fall under Fair Use?</p>
  <p> </p>
  <p>4</p>
  <p>Additional issues</p>
  <p>7. Are there issues or chal enges that arose during your example that I haven’t asked you about?</p>
  <p> </p>
  <p>8. Who else should I interview? I am trying to reach big social researchers, qualitative researchers who have published or reused data, and data curators who have worked with qualitative or big social data.</p>
  <p> </p>
  <p>5</p>
  <p></p><h1>Document Outline</h1>
  <ul>
  <li><a title="#p1">Big Social Researchers Interview Guide</a>
  <ul>
  <li><a title="#p1">Project Overview</a>
  <ul>
  <li><a title="#p1">Project Title</a></li>
  <li><a title="#p1">Research summary</a></li>
  </ul>
  </li>
  <li><a title="#p1">Research background</a></li>
  <li><a title="#p2">Interview questions</a>
  <ul>
  <li><a title="#p2">Introductory question</a></li>
  <li><a title="#p2">Identifying a specific example</a></li>
  <li><a title="#p2">Context</a></li>
  <li><a title="#p3">Data quality</a></li>
  <li><a title="#p3">Data comparability</a></li>
  <li><a title="#p4">Informed consent</a></li>
  <li><a title="#p4">Privacy and confidentiality</a></li>
  <li><a title="#p4">Intellectual property</a></li>
  <li><a title="#p5">Additional issues</a></li>
  </ul>
  </li>
  </ul>
  </li>
  <li><a title="#p5">8. Who else should I interview? I am trying to reach big social researchers, qualitative researchers who have published or reused data, and data curators who have worked with qualitative or big social data. </a></li>
  </ul><p></p>


name: BSR_01
filename: Mannheimer_BSR01_Tra.pdf
created: 2024-06-21 19:03:59.295136
text_direction: LEFT_TO_RIGHT
contents:
  <p> BSR01_transcript_deidentified</p>
  <h2><b>SUMMARY KEYWORDS </b></h2>
  <p>data, people, platform, project, reddit, research, questions, big, paper, api, user, context, moderators, debates, posts, review, repository, gender, community, findings <b>SPEAKERS </b></p>
  <p>BSR01, Sara Mannheimer</p>
  <p><b>BSR01  </b>00:00</p>
  <p>I haven't looked at the questions you sent. Yeah, but this is an active project. So I don't think I should have a problem with any of them. Okay, great.</p>
  <p><b>Sara Mannheimer  </b>00:13</p>
  <p>And do you have? Do you want to follow along? I can resend the email that had the attachments, if you like. That may help you.</p>
  <p><b>BSR01  </b>00:23</p>
  <p>I have the email. Yes. Let me just open it up.</p>
  <p><b>Sara Mannheimer  </b>00:26</p>
  <p>And you don't have to I can also just speak, but it's sometimes nice to have it to read as well, since some of the questions are more complex.</p>
  <p><b>BSR01  </b>00:35</p>
  <p>The big social interview dot, dot PDF.</p>
  <p><b>Sara Mannheimer  </b>00:40</p>
  <p>Yep, that's it. Yep. So hi, my name is Sara. I'm a librarian at Montana State University. And so the reason I'm doing this research is trying to figure out you know, big social data is a newer practice. And we have a lot of questions about data curation surrounding big social data. So the idea behind this project is that I'm trying to connect the communities of qualitative researchers and big social data researchers, and the data curation practices that are done in each community to try to connect the two and hopefully learn some things, maybe about scaling up for qualitative research, and maybe about better data management practices in big social research, and more ethical and epistemologically sound research. So, so yeah, let's get started. Can you tell me a bit about the type of research you generally do and what kind of data you produce?</p>
  <p><b>BSR01  </b>01:43</p>
  <p>Yes, um, so my research is, there are two types. I work on methodological research developing methods to analyze conversations, to do things like detect fraudulent reviews, fraudulent transactions, things like that. I also work on understanding human behavior in conversations using big online conversational data sets. And one of the projects I've been working on for a while is understanding conversations on this platform called [Online Debate Platform]. And I'm specifically looking at what is the impact of status indicators on persuasion on this platform. And various status indicators give you</p>
  <p>enhanced persuasive power, just by virtue of themselves. And also to what extent do status indicators motivate people to participate? Because it's a voluntary platform. From the platform's perspective, you want people to participate. But you also don't want them to have any special advantage. So I'm trying to tease apart: how do these two things go together? And what platforms can do about it.</p>
  <p><b>Sara Mannheimer  </b>03:06</p>
  <p>Yeah, the paper that I read was really interesting. I didn't understand all of the methods, so I'm glad you agreed to talk. So you've identified this specific example. And you're going to use the one with the</p>
  <p>[Online Debate Platform] and understanding behaviors there? Okay. So can you describe your data collection method from that platform?</p>
  <p><b>BSR01  </b>03:34</p>
  <p>Yeah. So there are two types of data I'm collecting from this platform, the first is just archival or historical conversation data. So this is like, it's data going back to [the 2010s], when the platform first started, to today. And it's like around mil ions of conversations. The way I collect this data is, there are three sources actually, the first is using this archiving platform called Push Shift. Or what Push Shift does is it archives everything on Reddit</p>
  <p><b>Sara Mannheimer  </b>04:14</p>
  <p>Is it push shift? S H I F T?</p>
  <p><b>BSR01  </b>04:18</p>
  <p>Yeah, Push Shift. So they archive everything on Reddit using... they figured out a trick, which is basically every Reddit post or comment ID is just basically an integer. It's a number incremented, one by one. So what they do is they get all of them and store them somewhere. Push Shift is nice, but it's it has a lot of holes, there are no guarantees that your data is complete, which I realized because I was reconstructing these conversation trees from the data and then I realized that some comments are missing. So a second data source that I use is the official Reddit API that's provided by Reddit. And it's like, you can give it a comment ID or a post ID, and it gives you back that specific comment. But there are a bunch of limitations, you can only access the API once per second. If you're looking for the most recent comments by a user, you'l  only get 1000 at most. So that's the second source. The third source is something called the Push Shift beta API. It's like the Push Shift API. But it's it's just, I don't know what exactly the difference is, it's just in a different location, and sometimes Has more, or some of the missing data is available in this source. So those are three data sources. And I collect them all using some kind of Python script that connects to the API, fetches the data and stores it on on my machine.</p>
  <p><b>Sara Mannheimer  </b>06:04</p>
  <p>Yeah, we'l  talk about missing data later on. And actually, why don't we just talk about it now, since we were you've talked about your like, strategies for pulling in missing data, we're also going to talk about data quality. So how you account for data quality? An example of when you considered that issue? So you've talked about getting missing data from the API's. And you've also talked about your methods being used to detect fraudulent posts or bots. But what other examples do you have about data quality issues in your research?</p>
  <p><b>BSR01  </b>06:46</p>
  <p>Um, so one example is missing data. Another example is, I work a lot with text data. So sometimes parsing the text data that's online, into something that's human readable. That could cause issues, especially when people use complex—these Unicode characters, basically, from different languages.</p>
  <p> </p>
  <p><b>Sara Mannheimer  </b>07:14</p>
  <p>Yeah.</p>
  <p><b>BSR01  </b>07:14</p>
  <p>When you try to read that, in many programming languages, they just, they're just removed, like those characters are just removed, and things don't make any sense. So that's a big issue with text data.</p>
  <p><b>Sara Mannheimer  </b>07:29</p>
  <p>What do you do to account for that? Is there a way for that you've been able to correct those mistakes?</p>
  <p><b>BSR01  </b>07:36</p>
  <p>In my specific project, I haven't needed to yet. But there is a way. Uh, you'd have to treat the text as this format called Unicode. It's a, it's a, it's a character set that can accommodate a bunch of languages and special characters. So I'd have to treat the text as Unicode and that should fix things. Okay. Yeah, it's a bit more effort from my side.</p>
  <p><b>Sara Mannheimer  </b>08:04</p>
  <p>And do you share the data that, I think this question is back where we were, but do you share the data that you pull down from the Reddit API? With your paper or in other places?</p>
  <p><b>BSR01  </b>08:18</p>
  <p>Yes, yes, I usually do, I usually share the data once the paper's published, or because then I'm like, I'm sure that the data is in its final form, and it's clean. And it also replicates the results in my paper. I have some.... So if you just, if you go to my website, the datasets for all my previous papers have been released.</p>
  <p><b>Sara Mannheimer  </b>08:43</p>
  <p>Okay. And do you... Where do you release them, just on GitHub?</p>
  <p><b>BSR01  </b>08:49</p>
  <p>I yes, I release them on GitHub. Sometimes. So GitHub is not good for fairly big data sets, which is what my data—it might change—my data is right now. So I am I'm trying to find a better place to share that data set. I might just share it on my website as a raw download. I'd have to find a good a better platform.</p>
  <p><b>Sara Mannheimer  </b>09:16</p>
  <p>Have you heard of Zenodo?</p>
  <p><b>BSR01  </b>09:20</p>
  <p>No, no.</p>
  <p><b>Sara Mannheimer  </b>09:20</p>
  <p>I'm not sure it's through CERN, which is a Particle Physics Laboratory. Z E N O D O. It's a data repository. I think they can accommodate big files, but I'm not sure. Maybe it's.... Yeah, I think you'd have to check with them. You might have to like have a special, some special agreement. But that's a good sort of just general purpose repository, if you're looking for something.</p>
  <p><b>BSR01  </b>09:38</p>
  <p>Yeah, there's that. Thanks for that. I've also heard of Harvard Dataverse. I haven't used these platforms. The issue with uploading stuff on these platforms is they don't show up on search results most of the times. So people won't stumble onto your data sets the way they would on Github. Kaggle, Kaggle is another place where I could upload it.</p>
  <p><b>Sara Mannheimer  </b>10:07</p>
  <p>That's great. That's a good idea. Yeah, from a library perspective, we're like thinking about preservation as a data. And so the data repository has a DOI. So it would potentially show up. If you're searching like Google, it wouldn't show up in Google Scholar. But have you seen Google data set search? That's a sort of.... I guess it just came out of beta. So it would show up there, but Zenodo and other like data specific repositories that are about preservation, so and you can actually connect Zenodo with GitHub to get a DUI for your repo. But anyway, this is the beside the point.</p>
  <p><b>BSR01  </b>10:53</p>
  <p>This is useful.</p>
  <p><b>Sara Mannheimer  </b>10:56</p>
  <p>Okay, and maybe [your institution], if you ask the librarians there, they might be able to help you with it, too. They might have a data repository locally.</p>
  <p><b>BSR01  </b>11:09</p>
  <p>Yes. Yeah, I should, I should check with them for sure.</p>
  <p><b>Sara Mannheimer  </b>11:13</p>
  <p>And so is your project grant funded? Were you required to share the data? Or is this just something that you're interested in yourself?</p>
  <p><b>BSR01  </b>11:20</p>
  <p>No, these are not grant funded. Yeah. These are just funded by my PhD stipend.</p>
  <p><b>Sara Mannheimer  </b>11:35</p>
  <p>Okay, let's move on to context. So I have this quote here. "When we collect data from social media platforms, context matters. But the context of social media might be absent or difficult to understand if you're taking short pieces of text out of the broader context of who the person is, what their personal and private life or in public life are. And then when the data are masked at a large scale that can be even more pronounced." So when I talk about context, that's basically what I'm thinking about. So can you tell me about a time, if any, during your process, when you considered the issue of how you might maintain and understand your data as context, both for you doing the research and for, potentially for users when you share the data?</p>
  <p><b>BSR01  </b>12:21</p>
  <p>Yes, yeah, context was... So when I submitted this, the first project from this data set to a journal, it was the conversations between users on [Online Debate Platform], but I was trying to answer a managerial question or a question or a question about human behavior in general. So I needed to qualify my findings with what is the population I'm looking at. I would have liked to know, things like what is the age, age group that people are in? What topics are they interested in debating? Why do they participate on this platform? Things that if I, if I could do a survey of these users, I would ask them a lot of questions. But I can't. And all the users are anonymous in the sense that there is no identifying information associated with their usernames. So I actually don't, I don't have stated context from the</p>
  <p>platform. The way I worked around this was to look at overall Reddit demographics. And assume that people on this part of Reddit on this subreddit would be similar to the overall demographics. That's, that's a pretty strong assumption. It was just kind of the best I had.</p>
  <p><b>Sara Mannheimer  </b>13:47</p>
  <p>And did you talk about, you know, when you publish your data, or you publish your paper, do you talk about that in the paper as like a limitation? Or do you have a discussion of it?</p>
  <p><b>BSR01  </b>13:58</p>
  <p>It's something it's not there right now. It's a, it's a comment that a reviewer brought up. So, we'l definitely... Yeah, it's going to be like a limitation or something that's going to be in the conclusion talking about the external validity of, of our findings.</p>
  <p><b>Sara Mannheimer  </b>14:15</p>
  <p>Okay. And then did you, besides the review, or after the reviewer brought it up, did you like consult with anyone else or look at other research projects? How did you come to your thoughts about how to present context and your data?</p>
  <p><b>BSR01  </b>14:31</p>
  <p>Yeah, I did look at other papers that use [Online Debate Platform] data, or that use Reddit data. I guess specifically in the journal that I was trying to publish in. Because in, in, so in CS, in computer science communities, it's not that important, because everyone just assumes that your findings are limited to some specific context. But in management journals, it's a lot more important. So you see people talking about the context.</p>
  <p><b>Sara Mannheimer  </b>15:07</p>
  <p>Okay. That's interesting. That's part of my, what I'm wondering is like how different communities, different disciplines relate to these questions. So that's super helpful.</p>
  <p><b>BSR01  </b>15:21</p>
  <p>Yeah, so I publish in both, in computer science and management. So yeah, I can I can talk about it.</p>
  <p>Quite a few differences.</p>
  <p><b>Sara Mannheimer  </b>15:30</p>
  <p>Okay, cool. Yeah, I have... So I meant to tell you, there are six different issues that we're going to go through context, data quality, data comparability, consent, confidentiality, and intellectual property. And so you'l  have more opportunities there to sort of explore it. These are issues that I identified in both qualitative data and big social data. Okay, let's, and there are eight questions total when we're on number three. So, during your example, during your project, did you compare or combine multiple big data sets? Or have you considered comparability or interoperability of your Reddit data set?</p>
  <p><b>BSR01  </b>16:12</p>
  <p>Um, I didn't. So far, I've not had to. So I collected the same data from multiple sources. I didn't have to combine it with other datasets, there was another project, where I was looking at biases in the text of peer review, where I collected data from an online conference review portal and I merged it with data from ArXiv, which is a paper archive repository. That was the only time where I needed to do this merging, but not in this present project.</p>
  <p><b>Sara Mannheimer  </b>16:54</p>
  <p>Okay. Well, we can I think we can discuss that time, we can just switch it out. What strategies did you use to combine the datasets? Did you have to do any sort of adjustment of the data format? Or metadata?</p>
  <p><b>BSR01  </b>17:08</p>
  <p>Yeah, it was pretty difficult. So I had papers on that were submitted to this conference called</p>
  <p>[Conference Name], that I got from this [online conference review portal]. I needed to match those papers to papers on ArXiv. Um, so the first chal enge is, how do I even find these, I could look for the paper title. Sometimes people upload it with different titles, to avoid reviewers finding it and then getting to know who the authors are because the conference was double blind, in, after 2018. Now, I'd have to do things like do a search for the abstract. Even the abstract could, would not match that some conferences have a policy that you can't use the same abstract in your public key visible version of the paper. So you have to do something like a fuzzy text match. The good thing about this data set was it was small, so I could manually inspect every single match to make sure that it's right. So I could check for false positive matches, but not for false negative match. And so if I did not find a match, I didn't actually go and search for it manually.</p>
  <p><b>Sara Mannheimer  </b>18:25</p>
  <p>Okay. Yeah, that's, this is a tricky one, especial y the bigger the data gets, it's harder. I feel like there's a lot of potential there. But without common data standards that everyone's using, it can be really challenging.</p>
  <p><b>BSR01  </b>18:44</p>
  <p>For bigger data sets, I usually plot simple statistics, like for [Online Debate Platform], I just plotted the number of comments and the number of posts every week, then that's what that's when I noticed that in February, there were like, a 10th of the amount of normal amount of posts, and that's when I knew that there was missing data.</p>
  <p><b>Sara Mannheimer  </b>19:04</p>
  <p>Okay, nice. Great. Okay. Let's move on to question four about informed consent. Can you tell me about a time during your project that you considered informed consent?</p>
  <p><b>BSR01  </b>19:21</p>
  <p>Yes, yes. Yeah. Yeah. So the [Online Debate Platform], I guess I didn't talk about it. I said the the first part of machine learning project was using historical data. The second part is a using a randomized experiment. So what I'm doing right now, what I just did, three months back was, I spoke to the moderators of the platform, who gave me permission to hide the reputation of anyone on the platform.</p>
  <p>And that way, that way I could do an experiment, I could randomly select 50% of the users hide their reputation and then observe their participation and their success, what successful persuasion rate over the course of three months. And then I can measure if there were any differences. And if there were differences, I could, I could attribute them to hiding reputation because their reputation was hidden randomly. That's what I did. I had to.... so before I started this experiment, I did have to get approval from the [Institution] IRB. And the IRB specifically asked me a lot of questions about consent. So they were interested in firstly, are the people on the platform going to know that you're hiding their reputation? And for me, it would be bad if they knew because that would change their behavior. So I didn't want to explicitly tell them. So I had to justify that this was not, not having informed consent while doing the experiment was not causing a lot of harm.</p>
  <p><b>Sara Mannheimer  </b>21:07</p>
  <p>Okay.</p>
  <p><b>BSR01  </b>21:09</p>
  <p>So that's, since this is a voluntary debating platform, and people are really participating, to get points and get better debating. I argued that there is there is no real harm in this, in this setting. So that was fine. That that worked for the IRB, though people did...</p>
  <p><b>Sara Mannheimer  </b>21:38</p>
  <p>Yeah, go ahead.</p>
  <p><b>BSR01  </b>21:40</p>
  <p>So people did, once I hid people's reputation, they did eventually find out. Because you kind of noticed, especially if you're an active participant. But yeah, and...</p>
  <p><b>Sara Mannheimer  </b>21:52</p>
  <p>Did they tell the moderators that they hadn't seen that? How did you know that...?</p>
  <p><b>BSR01  </b>21:56</p>
  <p>One of them did, one of them asked the moderators what happened, um, but I also saw that the people who I hid the reputation for—they reduced their participation, and the people who I did not try to hide the reputation for did not change.</p>
  <p><b>Sara Mannheimer  </b>22:14</p>
  <p>So so it did affect their, it affected their standing in the community, maybe like maybe people weren't seeing their posts, or....</p>
  <p><b>BSR01  </b>22:22</p>
  <p>It also affected, so it had a dual effect. It affected the number of debates they participated in, in those three months. So the people whose reputation was hidden, participated in on average, to fewer debates than the people in the control group. It also once you control for the number of debates, it also affected their success rate. So how many successful debates they had given a certain number of debates, they participated. So it had a dual effect on both of them.</p>
  <p><b>Sara Mannheimer  </b>23:00</p>
  <p>So did you discuss like, Do you have plans? Or did you do like, what's your relationship with the moderator? Is, I'm wondering, like, if they could give you information about who they think their people are, what kind of consent they expect, or you know, what these what the research might, how it might affect them? You know?</p>
  <p><b>BSR01  </b>23:18</p>
  <p>Yes, yeah, I'm so the, I created a Discord channel, to be in touch with one moderator who sort of my point of contact with the rest of the moderation team. I think they have around eight or 10 moderators.</p>
  <p>And I keep sharing my preliminary findings with this moderator and are getting feedback, talking about how this could help the platform. They seem pretty interested in the findings. And the the plan is once I write this up, they share it with the rest of the community.</p>
  <p><b>Sara Mannheimer  </b>23:53</p>
  <p>Okay. Is there... So did you review the Reddit Terms of Service?</p>
  <p> </p>
  <p><b>BSR01  </b>24:01</p>
  <p>Yes, yeah, yes, yes. Yeah, I reviewed them specifically for the case of col ecting data. So they have certain terms, to use their API, things like that. I did not review them before doing the experiment. And that is because I had read quite a few papers that did their experiments. And I just assumed it was fine.</p>
  <p>I didn't get I didn't actually look.</p>
  <p><b>Sara Mannheimer  </b>24:33</p>
  <p>Yeah, I'm not sure what their terms of service say either. Like I think that Facebook does specifically say when you sign up for the platform, you might get research done on you or with your with your data.</p>
  <p>Okay, cool. Al  right on to privacy and confidentiality. Can you tell me about a time if any during your research process when you considered issues of privacy, protecting the data during your research or protecting the people on the platform.</p>
  <p><b>BSR01  </b>25:03</p>
  <p>Yeah, so a good example of that would be my previous project where I was looking at biases in the language of peer review. I was looking at gender bias, what are the biases looking with gender bias.</p>
  <p>For that we had an external annotator annotate gender for the authors on the platform of these papers.</p>
  <p>And the protocol we followed was semi automatic. So we use the US Social Security data to infer genders with some confidence probability. For example, if there was a certain name, let's say Jack, that was reported as male 99% of the time, we would tag it as male. And same thing if it was reported as female and 99% time we would tag it as female or non male rather. But if it was less confident, we had the human annotator, search for the name search for their Google profile and make a guess as to what this person's gender is. So we were, so we were trying to mimic the reviewers perception of the author's agenda. And because bias, the bias would be driven based on that perception, and not the self reported gender of the author. We did not release these gender annotations, because we didn't, we thought it would cause issues if we mislabeled an author's gender. And these authors are a part of the community that we publish. Yeah. So we opted to not release identities. And we actually at the end of the paper, we actually have an ethics statement where we talk about why we're not doing that. But the data, the data itself is public.</p>
  <p><b>Sara Mannheimer  </b>27:08</p>
  <p>And I guess sometimes with social media, there are like considerations of move it, like moving the data out, out of its context, in Reddit, you know, and you're letting it be downloaded. I think in your situation, since all of the users were anonymous, that changes things a little.</p>
  <p><b>BSR01  </b>27:33</p>
  <p>So they were anonymous in the sense that I couldn't tie it to their real identity. But some people use this same username on all these online platforms. And that is that that is a concern. So let's say there's a user in my data who said something they regret, and they delete their post or their comment. It's stil going to be there in my data set.</p>
  <p><b>Sara Mannheimer  </b>27:57</p>
  <p>Right.</p>
  <p><b>BSR01  </b>28:00</p>
  <p>And that is, that is an issue that I have thought about, but I haven't figured out how to deal with it yet.</p>
  <p><b>Sara Mannheimer  </b>28:09</p>
  <p>Yeah.</p>
  <p> <b>BSR01  </b>28:12</p>
  <p>I know that Push Shift has on the, Push Shift has a subReddit, where people can submit data deletion requests.</p>
  <p><b>Sara Mannheimer  </b>28:22</p>
  <p>Okay. Interesting. So yeah, one of my questions is, did you feel that you had to make any compromises to privacy in order to conduct your research? Like, is this a major concern to you? Or do you feel it's small enough that it's not compromising user privacy?</p>
  <p><b>BSR01  </b>28:43</p>
  <p>So it's not a compromise to my research. It's a compromise to replicating my research. So if, if I don't release the data, it wil  be private. But then no one can replicate my results. It's going to be really hard because you need to col ect all this data again. Yeah. So that's the trade off that....</p>
  <p><b>Sara Mannheimer  </b>29:13</p>
  <p>Yeah, like weighing the benefits of people being able to replicate and reuse your, replicate your research, and reuse your data, and then the potential privacy implications for the user. Cool. Great, great questions to be asking. Al  right. This is our last major question about intellectual property. Was there a time during the process of this project where you thought about intellectual property concerns of the participant or of the platform for anyone else?</p>
  <p><b>BSR01  </b>29:53</p>
  <p>Um, so the, these two projects which I just spoke about peer review and [Online Debate Platform] didn't have any intellectual, I didn't have I didn't think about intellectual property concerns.</p>
  <p><b>Sara Mannheimer  </b>30:09</p>
  <p>That's a good answer too. And then, are there any additional issues or challenges that arose during your example that I didn't ask you about or something you want to talk more about?</p>
  <p><b>BSR01  </b>30:28</p>
  <p>So I think one, if one thing about missing data is the data is missing, because it was just wasn't archived. There could also be data that's missing because the authors deleted themselves from the platform. Okay. And that usually doesn't show up as missing data, it shows up as kind of text that is not associated with any author. So on Reddit, it shows up with the username of deleted.</p>
  <p><b>Sara Mannheimer  </b>31:08</p>
  <p>Oh, okay, but the comment is stil  there?</p>
  <p><b>BSR01  </b>31:11</p>
  <p>The comment is stil  there. And now I needed to think about what do I do with these comments? Do I add them in and treat them like they were made by a known, an unknown user? A unique unknown user? But then what if like, 100 of them are made by the same user, that would change my finding. So what I did in my project was to just drop all deleted comments. And also show that the results don't change much if you include them. Okay.</p>
  <p><b>Sara Mannheimer  </b>31:44</p>
  <p>That's really interesting. And all of this, and you'l , and you talk about that in your paper as well. In your methods.</p>
  <p> <b>BSR01  </b>31:53</p>
  <p>Yeah, right.</p>
  <p><b>Sara Mannheimer  </b>31:54</p>
  <p>Yeah, yeah. Okay.</p>
  <p><b>BSR01  </b>31:58</p>
  <p>Um, yeah, I think I think I've spoken about pretty much everything about this data.</p>
  <p><b>Sara Mannheimer  </b>32:09</p>
  <p>Okay. This is really great. Thank you so much. Okay. I really appreciate it. Well, thanks so much for your time. And good luck with this project. I can't wait to read the next steps. It seems so cool.</p>
  <p><b>BSR01  </b>34:14</p>
  <p>Yeah, thanks. Yeah. Good luck with the rest of your interviews.</p>
  <p><b>Sara Mannheimer  </b>34:18</p>
  <p>Thanks so much.</p>
  <p><b>BSR01  </b>34:19</p>
  <p>Right, bye.</p>
  <p> </p>


name: BSR_02
filename: Mannheimer_BSR02_Tra.pdf
created: 2024-06-21 19:17:31.832312
text_direction: LEFT_TO_RIGHT
contents:
  <p> BSR02_transcript_deidentified</p>

  <h2><b>SUMMARY KEYWORDS </b></h2>
  <p>wikipedia, people, data, documented, context, paper, database, community, big, conflict, metadata, interviews, bots, librarians, project, method, cases, dumps, feel, public <b>SPEAKERS </b></p>
  <p>Sara Mannheimer, BSR02</p>
  <p><b>Sara Mannheimer  </b>00:01</p>
  <p>Alright, so basically the project I'm working on is about connecting two communities of practice, using strategies from qualitative data curation to support big social research. So my hope is that by connecting a community that's more established and thinking about complex issues around data sharing with a community that is newer and hasn't done as much thinking in that regard, we'l  be able to maybe use some of the data curation strategies that we've developed for qualitative data sharing, to support big social data sharing, or big social research as wel . So I'm talking to big social researchers, I'm talking to qualitative researchers who have shared their data and I'm talking to data curators, with the goal of developing some strategies that librarians and data curators can use to support these two types of research. And so I did a lit review where I identified six key issues that are challenges for both types of research, context, data quality, data comparability, privacy and confidentiality, informed consent and in-, intellectual property. So this interview wil  be structured around those six topics, and I'l ask a question for each of them. And I have a intro and a wrap up question as well. So we'l  have about seven, seven questions, eight questions. And it should take about an hour, we'l  see. Okay, you ready to start? Okay. So tell me about the type of research that you do. And then what type of data that you produce generally.</p>
  <p><b>BSR02  </b>01:44</p>
  <p>Okay, um, I do too many different kinds of things for my own good. I very much a kind of disciplinary nomad, I kind of have, you know, I feel like one foot in, in in everything, which sometimes leaves me like, I'm on a on a on a chasm, but I am in my, my current, I don't know. So I'm currently in a joint appointment between the [communications department], which is in the social sciences, and the [data science unit], which is [engineering focused]. And so I am literally, in between these two fields. When I'm when I kind of have my, my kind of communication hat on and my kind of qualitative social science head on I am often more of a, someone who does interviews and who does ethnography, I was trained as an ethnographer, to doing a lot of like, in person participant observation of studying how scientists work. And so I've done a lot of kind of actually similar kinds of topics about like, how do scientists like and I've worked with data, especial y, it's been a lot of time studying ecologist and in my grad program, and how they were, you know, trying to do kind of global ecology and data integration and things like that. I've also spent a lot of time with a more kind of decentralized communities of knowledge production and dissemination. So I've done a lot of work on on Wikipedia, and how Wikipedians col aborate, and how kind of conflict emerges and how they resolve conflict. I've done some work on on Twitter on how like social people who are users of Twitter, I've done, did a project on on people who have been harassed on Twitter and the subject of kind of coordinated harassment campaigns, and how kind of their experiences and so I feel like I could have sometimes my, when I'm on the more like social science of things, my methods are often more, you know, interviews, observations, qualitative kind of</p>
  <p>smal  data kind of stuff. But I have done more work that has led me into the, I guess, more of the big social data space, especially when it comes to, and those have often been in collaboration with people who have more of the computational skil s. And so I so I would say, the most were the work that I've done with more big data methods has been on Wikipedia, where the data is open and accessible. And so we've done a lot of things on like, I've done some stuff on, like, what are newcomers experiences, like when they join when they first start contributing to Wikipedia? And so in that case, you can, you know, there's like interview methods, and then like, opposite, like more like just reading of the archives in the forums, and, you know, looking at how using kind of more traditional humanities inflected kind of methods and theories. And then I've also done more kind of trace data analysis of, you know, so like looking at, like, who are the people who actually stay and leave, we can sort of segment those people computationally. We can see if, you know, did did risk doing kind of more more things where there are p values, you know, where it's it's regressions of, you know, did this thing happen to a newcomer and did that, you know, does that predict whether they continue to contribute or whether they stopped contributing.</p>
  <p><b>Sara Mannheimer  </b>04:57</p>
  <p>Hm, okay. I'm wondering if I should interview you as a qualitative researcher. So I think, it's actually really perfect for me. Because I, like I have sort of seeing that these two communities aren't connecting as much as they could. So I feel like you're the perfect person to talk to you about.... Okay, so we'l  try and like, yeah, I wil  see, let's talk about a big like....so the way I'm doing this is using critical incident technique, it's called. So using like, one specific time that you, one specific project, it can kind of help get it some more detailed specifics. But you can also feel free to go beyond your critical incident if you need to, or if you want to. So let's first identify an example when you either collected big social data for research or used big social data or prepared big social data for sharing, or considered sharing your data. Maybe this Wikipedia example is a good one.</p>
  <p><b>BSR02  </b>06:07</p>
  <p>And I've got I've got the exact paper in mind that that was a big one. Yeah, I can share the link with you too, because we actually tried to go into a lot of methodological detail about how we did it. I've actually got, I can share with you this massive like, just to give you a scale of this, I'l  do a screen share real quick, but oh, no, nevermind. But yeah, I can, I can share it with you many ways... the project was around, this is actually a project we did that was a rebuttal of another study, that, in our view, was making a claim about Wikipedia, and they were trying to detect conflict and Wikipedia. And based on sort of my ethnographic understanding of what goes on in that community, I think that they were just absolutely wrong. And so, and we've been very public about this, it's a was a, you know, we, we made a lot of criticisms of how they were operationalizing conflict. And we thought that that sort of didn't align.</p>
  <p>And so we, we did this sort of big data, small data project, where we wanted to find all these cases where their method had identified these cases as conflict. And we did not think based on, you know, my ethnographic understanding of the situation....</p>
  <p><b>Sara Mannheimer  </b>07:23</p>
  <p>Oh, okay.</p>
  <p><b>BSR02  </b>07:23</p>
  <p>...that in the Wikipedia sense of things, like it was not conflict, because it was actually either cooperation or just like a non-event. And so what we tried to do is I worked with my, I was I was a lead author on the project, and I was working with my longtime computer science collaborator. And actually, we kind of flipped our normal roles that we did. So I was actually taking the lead on the Big Data part of things. And he did more like he did interviews with some of the people who were involved and kind of</p>
  <p>talked to them about their experiences of these events. And so we were kind of trying to, like invert that traditional like, we're like, typically, like, they do like the method first, and they did the findings. But we wanted to kind of start with these cases and kind of see where their method broke down. And then we wanted to then replicate, we wanted to kind of produce our own quantitative metric, or not metric, but like quantitative operationalization, where we can feed in all this like edit activity data about who does what, and like, who makes an edit to a page, and then who like, undoes that edit and replacing it with something else, and then put, who puts it back? And, and so we wanted to have have our own method for detecting conflict based on those kinds of those kinds of metadata and records. And so we did that.</p>
  <p>And yeah, we did a whole lot of work documenting it. And you know, it's something that goes through and can scrape through the entire ful  history of every change that's made every Wikipedia article in the multiple language versions that Wikipedia has. Actually, I think we only did five or seven language versions, but like, stil , like, it was a big effort. Yeah, you know, so that's, that's the context of this. We ended up making our entire data and pipelines, so not just the data, but also all the all the scripts, all the Python, all the Jupyter notebooks, all the builds, like that's all on GitHub and Zenodo. It's all well documented. I have this like, massive, ridiculous flowchart diagram of like, "Okay, we've got the, the Wikipedia SQL database that exports into this format, and then, you know, now it's in a XML format, and then we have the XML parser." And so I don't know, it's, it's, we actually have that pretty well documented, if you want to ever follow through on that later. So.</p>
  <p><b>Sara Mannheimer  </b>09:37</p>
  <p>Okay, yeah, I'l  take a look. Perfect. That's a great example. Okay, so your data collection method was scraping the web when you did this?</p>
  <p><b>BSR02  </b>09:50</p>
  <p>So thankfully, Wikipedia has their database in so they have a SQL database that they use for production, you know, that like that. Like the servers themselves that are hosting Wikipedia, you know, are structured in that way. And they regularly process actually I can I, at the time, it was someone named Ariel whose job it was to make I actually have this al  I've got this this thing that I've got my documentation up next. And so they regularly export that database to XML. And they publish those to the web. So we just could download the XML documents.</p>
  <p><b>Sara Mannheimer  </b>10:25</p>
  <p>Nice. Okay. Perfect. And did you have grant funding for the project that required specific treatment of the data?</p>
  <p><b>BSR02  </b>10:34</p>
  <p>It was, so we didn't receive any money from Wikipedia, but because the data comes from Wikipedia, that data is is Creative Commons licensed. And so you know, we have to, you know, continue to, yeah, so it's like in in our processing of it. We released that under the same license. And I also was funded under the so... I was at [Institution] at the time, and that was funded through a grant by [Funder]. And so they have a, an open access for the for the for the publications, but not for the data. So the publication is, is open access, because it was funded by this grant.</p>
  <p><b>Sara Mannheimer  </b>11:24</p>
  <p>Okay, cool. And did you have a data management plan for the project?</p>
  <p><b>BSR02  </b>11:31</p>
  <p>No, I'l  just be honest. We didn't have we didn't have a DMP, like we had, I would say we had plans about data management, we did not have a DMP. If that is a distinction that makes sense.</p>
  <p> <b>Sara Mannheimer  </b>11:45</p>
  <p>For sure. For sure. Alright, so you didn't have written plans, like a written policy document or anything, but you knew what you're planning to do?</p>
  <p><b>BSR02  </b>11:55</p>
  <p>Yeah, we've worked with this data before. So yeah, we were we were pretty ready to do that.</p>
  <p><b>Sara Mannheimer  </b>12:01</p>
  <p>Great. Okay, so you, and you published in Zenodo and GitHub, right?</p>
  <p><b>BSR02  </b>12:07</p>
  <p>And we put it everywhere. I think it's on there, it's on, The California Digital Library has its own? Yeah, lots of copies, keep stuff safe.</p>
  <p><b>Sara Mannheimer  </b>12:18</p>
  <p>So okay. Are you, do you have information science background?</p>
  <p><b>BSR02  </b>12:23</p>
  <p>Mm hmm, I'm a mutt so yeah.</p>
  <p><b>Sara Mannheimer  </b>12:28</p>
  <p>Like you talk like a librarian.</p>
  <p><b>BSR02  </b>12:31</p>
  <p>Some of my best friends are librarians actually, the [data science unit where I worked was situated in the] library. Oh, no, we actually, that was like the neutral spot on campus. And so I got to know a lot of librarians. My PhD is from a school of information, although we were one of the schools of information that like got rid of our library program, back in the '90s so.</p>
  <p><b>Sara Mannheimer  </b>12:51</p>
  <p>Okay, well, let's jump into these issues. The first one is context. So I have a quote here from that says,</p>
  <p>"Halavais suggests that when we collect data from social media platforms, just as when we collect data in traditional spaces, context matters. However, the context of a social media post may be absent or difficult to understand social media posts are by nature, short pieces of text, taken from a larger context of personal and public life, this out of context effect is only compounded when data are masked at a large scale." So I guess in your case, these posts that we were working with are like people's edits, right? And like their interactions between each other as they're working on Wikipedia. So this is sort of where I'm coming from, in terms of context, just thinking of like, context in the social media realm, similar to context and like a qualitative space where it's like, there's so much that you don't know, and that you may not be able to document. So can you tell me about a time if there were any, during the process of your example, when you considered the issue of maintaining and understanding the data's context? So like contextual information about Wikipedians themselves, or contextual information about how you collected the data or who you are and who your collaborators are? And then what were those considerations and how did you address those issues?</p>
  <p><b>BSR02  </b>14:16</p>
  <p>Yeah, yeah, yeah, so context was the heart of this, because, you know, that was the point of this paper is that we felt that the original study had not kind of gotten the proper context, you know, and so we felt</p>
  <p>like they took a very kind of, I guess, data driven is the word for like to use maybe, but you know, you know, kind of came from that, that that approach. And, and so we wanted to kind of put the context back in. So that was a central part of this. And part of that was that you know, both me and my collaborator, you know, I I'm more heavy on the qualitative humanities side. He's been trained as a computer scientist, but we both have had more than a decade of experience in Wikipedia. Me more from like an ethnographic perspective, we've been a long time kind of studying this project, he just as a volunteer contributor, and you know, was just very always involved, and then, and then made it a part of like his dissertation work as well as a computer scientist, and then actually was then working at the Wikimedia Foundation kind of doing, you know, when we, after his after his PhD, and so both of us had a kind of understanding of the culture of Wikipedia, and the culture of interactions between between people. And also like, between, there's a whole, like, part of the paper of a lot of people really centers on automated accounts on like bot accounts. And kind of, because those are often and so one of things that if you don't, if you like, like one of the things that if you don't know a lot about how those accounts work, you might like it when people hear bots, and they think, "Oh, it's like spam bots," or things like that. But one of the things that I know a lot about, I spent my dissertation writing about it, but it's about all these, like pro-social bots that are very, like, authorized by the community. And they're, there's a lot of debates that happen, like some of them, you know. And so they do kind of like a lot of like, routine work maintenance work, kind of finding replaces cleaning stuff up. And, you know, integrating data, and there's all kinds of community processes around approving those and raising issues when they look like they're messing up or going outside their scope of approval. And so we really knew a lot about that.</p>
  <p>And so we wanted to make sure that we were treating that with the proper context, and that we understood that, you know, if there was a particular pattern of interactions, you you need to actually look at the content of what those interactions were, and you needed to also look at, you know, who the who these accounts were like, if they were bots accounts, we need to treat them a little bit differently.</p>
  <p>And if there's certain kinds of like, and so there's certain things about like, the Wikipedia bureaucracy that I feel like we we knew about, and so we were able to bring that to bear. But to be honest, like, it's not data management for that is, like, hard, and I don't think I think we kept a lot of that in our heads.</p>
  <p>And I feel like we were much better about documenting everything we were doing on the kind of quantitative side of things. But you know, we did, we did sort of do a lot of work with Jupyter Notebooks, and, like, starting off with the quantitative sort of approach, where we would just use like, their, their metric and their their sort of method for identifying conflict, cases of conflict. And so we'd throw those in. And we had a Jupyter Notebook where we could just like, look through all these, and then we could sort of filter in, and we could sort of look through and and I could sort of print the full context of an interaction and say, "Oh, I don't think that's right." Or "I think that is right, I think I think their method captured this one, this seems like real conflict to me, this doesn't seem like real conflict to me." And then we could then like, go back up to the top of the notebook and like fiddle with some some things in the in the in the kind of the programming language, and then run it again and see, oh, does that exclude the thing that we want excluded? Does it keep in the thing that we want kept? And so that was kind of our process of like, putting the context back in. And then we saved all those notebooks. And those are available.</p>
  <p><b>Sara Mannheimer  </b>18:08</p>
  <p>Nice. That's so interesting. Okay, and did you think about like, this is coming up about comparing data comparability, but like, I'm thinking about previous studies that you and your collaborator had done with Wikipedia? Did those come into it at all? Or was, did you just use the knowledge that you had about context? Like, when you're publishing this data, would you connect it to other articles you've written and say, here's how we know that, that Wikipedians conflict looks like this for Wikipedia?</p>
  <p><b>BSR02  </b>18:46</p>
  <p>Yeah, it was, it was largely, I think we wanted to especially for this project, I think we want to make it also the kind of thing that you could give, like, it was like, a bit pedagogical and kind of walking these through from the beginning. So I think we did a lot of that just kind of from scratch, even though we could have. But actually, I think though, I think this paper was like more of a critique of a lot of the other prior work that even maybe we had done, that that didn't, you know, that was more based on just, let's just do the metadata. Let's not look at content, let's not look at context. Like it's just that that's just harder. And we've, we've published a lot more papers, and we've published other papers that are just, you know, just kind of counting things with metadata and kind of looking at trends up and down and not not looking at too much about the contact, content of what's actually changed. So.</p>
  <p><b>Sara Mannheimer  </b>19:41</p>
  <p>Okay, let me ask you this last question that's on the bullets here. Since you published your own data here, what strategies did you then use when you were publishing to communicate the context of your data to future users, you know, so you'd gone through the you know, process of the study that you were critiquing and redoing. What about for yours? Like, if someone's trying to reuse your data? What strategies did you use to make sure that they understood where you were coming from?</p>
  <p><b>BSR02  </b>20:12</p>
  <p>Yeah, so we have, I actually want to, like, refresh my memory here, and like, I'm going to the repository itself. Let's see, we have, yeah, no. So we do a pretty like this is I'm actually pretty proud, like, I led the effort on the data and, so I'm first author on that paper, right. And so I'm going through just this documentation file. And, you know, we talked about each of the data sets and how we generated them, and really made it so that like, you know, if you're going to rerun our project from scratch, like you'l  get the same, like, it's reproducible in that sense. And talking about how it's sort of getting all these different stuff from different places. And then and then I don't know, so actually, we have, it's more of it's more of like just giving documentation about like, what each like, like, I have a big table of our data, our data data dictionary table, to kind of say, like, okay, here's the field name. Here's the like, description of like, how we got there, and like a little bit of context, like, where in the pipeline was it created. And then some examples of each of those. And if like, that kind of captures, because like, some of these are ones that are like, straight from the database. And you know, they're they're, if you download the raw XML dumps, you'l  get them. But then some of the if it's like, ah, no, this is, this is a metadata. This is like another metadata calculated field that's like generated in this script. This is our, like, we do a lot of stuff of like, we built the parsing engine for like parsing the actual text of certain things. And so it's, you know, here's, here's our, like, calculated type of interaction of like conflict or not in conflict, here's where that comes from. But real y, I think we were imagining that the data documentation itself, and like, basically everything that we upload to Zenodo, and Figshare, and all the good places. I don't think that stands alone. Like, I feel like you need to read the paper. And we open the paper with like, a vignette of a case that and we like, present it, you know, like, I feel like you really need like, and I don't know if you put like a warning in that, but I feel like I was not, I don't think that if you read everything that's in Zenodo, you'l  get any of that context, you'l  just get a reproducible pipeline to like, I want to get the context, you need to read the paper, and I don't have any, like, I think I actually tried this a little bit. And I was just like, I can't figure out a way to do this. Like, if you want to get the context, read the paper, because the paper is everything about how we got the context into the data. You know, if that makes sense.</p>
  <p><b>Sara Mannheimer  </b>22:52</p>
  <p>Right. For sure. Yeah.</p>
  <p><b>BSR02  </b>22:53</p>
  <p>So if there's any, so maybe, maybe your research might figure out some ways to like, help us with that.</p>
  <p>But I think I actually just remember struggling with being able to do that. And I was just yeah, forget it.</p>
  <p>I'l  just, you know, the paper wil  stand out.</p>
  <p><b>Sara Mannheimer  </b>23:05</p>
  <p>Yeah, for sure. Huh. Okay, side note, what motivated you to go to all this time and effort to make sure that your data was so well documented and available?</p>
  <p><b>BSR02  </b>23:17</p>
  <p>[Detailed explanation redacted at request of participant. Summary explanation below at 27:02].</p>
  <p><b>Sara Mannheimer  </b>26:39</p>
  <p>Okay, cool. Yeah, well, I guess part of what we wonder as data librarians is like, What does motivate people to like, do a good job, because it's not real y like to do a good job of data sharing. Because there isn't that much external motivation right now. It you know, it's like, as long as you share, and you show that you've shared and shared, that's like, kind of all that seems to matter. So, <b>BSR02  </b>27:02</p>
  <p>Yeah. In a way that I'm more comfortable sharing, which is that I was contesting a paper that was widely circulated, got widespread coverage in in mass media, was led by a professor at [a well-regarded university], who led a prominent group, led by a person who has given keynote addresses at major conferences and [these conferences] were continuing to celebrate his work. We were contesting that. The stakes were very high. So we wanted to make sure we got it right. And we wanted to make sure that people had trust in our work. And that we knew that this might be something where we had a bunch of back and forth, because they also didn't document their methods sort of as well as we would have liked. And so we wanted to make sure that everything was as documented as we could, because it was going to be a matter of kind of contestation and debate.</p>
  <p><b>Sara Mannheimer  </b>27:52</p>
  <p>Okay, okay.</p>
  <p><b>BSR02  </b>27:53</p>
  <p>That's for the record.</p>
  <p><b>Sara Mannheimer  </b>27:56</p>
  <p>Great, let's move to data quality, I want to make sure, try and get us finished.</p>
  <p><b>BSR02  </b>27:59</p>
  <p>Okay, we can a few minutes more. Yeah.</p>
  <p><b>Sara Mannheimer  </b>28:02</p>
  <p>Okay, so tell me about a time during the process of your Wikipedia work when you consider the issue of data quality. So you kind of talked about like their misunderstanding of bots. But I guess you could also talk about if there were problems with data quality in the Wikipedia database, or if there was any other quality of your own method and how you addressed that or how you communicated the quality of your method when you ended up publishing the data later?</p>
  <p><b>BSR02  </b>28:41</p>
  <p>Yeah, it's a good question. I'm, I'm maybe struggling to think of the specifics. This was a few years ago.</p>
  <p>Definitely, there's a lot of stuff. I don't know, this is the kind of thing when you're working with text, like encodings and Unicode are just very frustrating. So there would often be times when like, something didn't work, because we're converting from so many multiple formats and encodings that sometimes, you know, you get, and also like people, you were dealing with multiple languages, which don't all have just the standard Latin character sets. So yeah, I remember I remember just Unicode being something that I was fighting a lot over in terms of that, in being able to get that in. I feel like I trust the Wikipedia data. Oh, ha! I got an example. We don't trust the Wikipedia database before 2004. We had, Wikipedia exists from 2001 to the present. In 2004, they did a massive redesign and it went from being something that was like very janky and very, like a very like old platform. And they kind of modernized and so they they there is so you stil  have the Edit history that goes back for like some of the older articles that goes back to 2001 by just after something that like, you know, in the Wikipedia, computational research community, we've talked about it before, like, I know people at the foundation who make the dumps.</p>
  <p>And so it's just sort of one of those things that, that we know that we cannot be confident at anything, the database says before 2004, before a certain time in 2004, when the new migration happened. And so we actually just sort of limited, we're like, we don't care about those early days. And so we can just start when the new database, we cut off our data set, I believe, starting when the transition happened.</p>
  <p>And so that's that's the kind of example that like, I know that from just my participation in the rather tight knit, like Wikipedia research community, you know, there's, there's like a comp, there's, there's a wiki for it has its own wiki, of course, and people talk to people document all that all this kind of stuff around, you know, the database was upgraded, has a new format. And so that changes how the XML dumps get processed. So is that the kind of thing that maybe is an example?</p>
  <p><b>Sara Mannheimer  </b>30:56</p>
  <p>Yeah, definitely. Yeah. And then I'm also, I think a lot of what you talked about with context is relevant here, too, like documenting your data really well, to make sure that future users understand its quality, and the quality of your scripts and al  of that. So I think we can apply those here as well.</p>
  <p><b>BSR02  </b>31:17</p>
  <p>Yeah, we also use a couple of like, some of our fields in our kind of model that are not our model or kind of classifier that we built for detecting conflict, non conflict require imputed metadata forms, like, like, how many bytes were changed between one version of a page and another version of the page, and the characters were changed. And so we, we calculate those, and then we store those in a separate kind of table. And so in our doc, we definitely have, you know, this is where this calculated field comes from. This is actually like, this script comes from there. If you want to poke and you want to change how we, you know, in calculated those fields, you can you can do that, if you don't, you know, trust us to make those, those those or you want to do it a different way. So that was also something that was important for us.</p>
  <p><b>Sara Mannheimer  </b>32:03</p>
  <p>Okay, great. Let's move to data comparability. So during your example, did you compare or combine multiple big social data sets? Or did you consider comparability or interoperability of your data set?</p>
  <p><b>BSR02  </b>32:20</p>
  <p>Good question. Um, I don't, we did have to combine from a couple different Wikipedia sources. So I'm actually just looking at this in terms of our big table. We have one big, which is like the entire edit history of everything that's happened to every article on Wikipedia. And then because one of the things that was very important were these like bots, and different bots do different kinds of things, we had another kind of database of, of, of bots, and like when they were approved, or if they, because like,</p>
  <p>actually, we did have some cases where like, someone created a bot, it got approved by the community, and then the bot developer, used their bot to do a whole bunch of stuff that they weren't supposed to do. And because they had a bot account, they could like edit every page in Wikipedia at once. And, you know, do a find/replace across the entire encyclopedia. People didn't like that, because they you know, so we have a couple of those that was like a more kind of curated list of, okay, we've got this big table of all these bots, and when they were, you know, and so we did have that that came from. And so we had to combine those we do have like, that's a little bit where we're like filtering out some of that, that definitely took some like hand curating. And we actually improved, we improved Wikipedia's own kind of internal table of how it manages that, as part of this project. We were like, we're doing this work anyway, why don't we do it on wiki data, which is this kind of relational database.</p>
  <p>Actually, that's not relational. It's a graph database that they're using for it was very new at the time.</p>
  <p>And so we just said, "Let's improve that version, and then build the pipeline that wil  query from Wikidata and the Wikipedia database." And so then we have our, our pipeline, just kind of, we made the edits to Wikidata data, the Wikidata database on the bot table, and then our pipeline just grabs from Wikidata instead of grabbing from our own like hand curated version.</p>
  <p><b>Sara Mannheimer  </b>34:14</p>
  <p>Got it. Okay, nice. That's a great example. Cool. Um, do you consult with anyone about like, how to support the comp-, the comparing between those two? Or was that just like you knew how it should work? And so changed it accordingly?</p>
  <p><b>BSR02  </b>34:34</p>
  <p>Yeah, I think we, I had to look up because Wikidata was new at the time, and it's kind of what they're using to kind of replace, you know, some older database forms. And so they were using it to keep track of, you know, which bots were approved when, and so I remember having to like read up on the documentation for that, but it was pretty well documented. And so I felt like it didn't I didn't need to ask anyone anything, because there was good documentation.</p>
  <p><b>Sara Mannheimer  </b>35:03</p>
  <p>Wikipedia is good as a data source.</p>
  <p><b>BSR02  </b>35:06</p>
  <p>Oh, yeah. Yeah.</p>
  <p><b>Sara Mannheimer  </b>35:09</p>
  <p>Al  right. Um, the last three issues we'l  talk about are more ethical and legal issues. So first, informed consent. Can you tell me about a time any during the process of your work when you considered informed consent?</p>
  <p><b>BSR02  </b>35:25</p>
  <p>Yeah, yeah. So we do consider the the database of everything that happens on Wikipedia to be public.</p>
  <p>There are issues around you know, I know this, like not all Twitter users know that Twitter is public, by default. There's kind of those issues that we think about. We do, because on Wikipedia, I mean, it's, we we tend to treat that more as public data. I've had, I've had IRB kind of determinations of non, you know, like, I forget the technical phrase, but I've, I've gotten one of the documents, it's like from our IRB, if it says, if you're working with the Wikipedia public data set, we don't consider that to be human subjects research. Even though that might be I don't know, personally, I might say that they may be a little bit looser on that than they should be, I don't know. But I do, I do think that because you do have to, like, every time you save an edit on Wikipedia, you actual y, you have to license it under a Creative</p>
  <p>Commons license. And so there is a like, you know, even if you edit from an IP address that wil  say, like, your IP, like, if you don't create an account, your IP address wil  be public. So they do provide a lot of like in the interface, I feel like it's more so than Twitter, in that sense. So I'm more comfortable treating that as a public data set. You know, we did do some member checking. And I do have an existing IRB that kind of covers, you know, so so I wouldn't necessarily, I wasn't sort of treating them in the same way that I was treating interviews. But I did have, like a very broad, existing IRB protocol that did cover the kinds of, like, interviews that me and my collaborator did around.... So basically, basically, what we did is we had these cases that were identified as conflict, and then we would give them to people. And then we would, and we did most of this actual y over email instead of interviews. But it was sort of more like, we wanted to have this in a way that we said, you know, "We're going to send this to you." And one of the things that we know we wanted to take care about was whether we could identify those people in the paper. And this is where Wikipedians are actually a bit different when it comes to like informed consent, where Wikipedia is it's a very public space. And so often, the informed consent is not around anonymity. But actually, like, they want credit for what they say, you know, it's like they want to get, and so like, de-identifying them is an act of, I don't know if I'd call it like an act of violence, but an act of erasure, it's an act, it's like, erasing their labor. And so you know, they mo-, and so we found that, actually, so we did double check, we kind of, we would have this interaction with them, we would say,</p>
  <p>"You know, we're writing this paper, it's covered under this, and you know, we're interested in this we want to use, we're interested in using your case, as an example. Can you talk about whether you thought this was conflict or not conflict? And then, you know, would you like us to, you know, actually detail this case in the paper," and we only used ones where the people said that they would be comfortable having their usernames and their case be described in our, in our vignettes and case studies.</p>
  <p><b>Sara Mannheimer  </b>38:26</p>
  <p>And most Wikipedia users use pseudonyms anyway, in their username, right? I feel like most people don't use their real name?</p>
  <p><b>BSR02  </b>38:35</p>
  <p>I would say like, the more you get to be a veteran in the community, it's more common, like so your username, people do use use pseudonyms for their usernames, but they'l  often then like, their if you go to their user page, their profile, they often kind of put a lot of information about themselves. Some don't. But like, yeah, some some remains, you know, very strongly pseudonymous, where they have a very strong identity, they say a lot about themselves, but they're you don't ever know, but but their actual name is beyond and they might just say, "Oh, I'm I'm an American," or something like that. But yeah, so so there is a culture where like, people are able to select the level of self disclosure on the platform. And, and so that I feel like especially the people we were talking to are much more longtime, sophisticated users, veterans who have then, you know, in these kind of we did, I don't think we did any, any newcomers, like, I think every case we did with someone who is very well established, and so I feel like has, has gotten the chance to set those out for themselves. And that's something that we also kind of know how to navigate that through our membership in the community. I actually was one of the very first when I was a grad student, like, started one of the first conversations about like researcher ethics and stil  have a page on like, how to ethically research Wikipedia that's on Wikipedia.  And that's like stil  used and yeah, so we think about these things a lot.</p>
  <p><b>Sara Mannheimer  </b>39:53</p>
  <p>Cool. Yeah, and your answer there kind of overlaps with the other two issues that we have, which is privacy and confidentiality and intellectual property. So for privacy, we've kind of talked through this idea that you, Wikipedia users may not want their information to be private, they may like expect to be</p>
  <p>public. Was there anything else about privacy that you thought about or that you talked about with users?</p>
  <p><b>BSR02  </b>40:26</p>
  <p>Not really. I mean, they were they were, I mean, to be honest, kind of very thin interactions that were mostly it over kind of, you know, text as opposed to interviews, we actually did have, you know, we actually found because this this, this original article got a lot of mainstream media coverage. There already was a conversation that was happening, and one of the candidates threads inside one of the community discussion spaces about it, and other people who were saying like, "This is ridiculous, these things seem wrong. I found some cases that don't apply." Yeah, so there already was a lot of like, public, internal, and all those discussions are public into the web. There's a very strong norm against having private discussions in Wikipedia. Everything is supposed to be done on the wiki, or the public chat rooms and things like that, mailing lists. And so yeah, so because of that, I feel like we were we were just in a different ethos. But we did give, because we did actually have an experience, this was like years ago, where there are some Wikipedians, who don't want to be identified and don't even want to be identified in...like, some people create tables of like the most active, like the people who have made the most edits to Wikipedia. And there's a few people who just don't even want their usernames to be on that list. Some of them like hold ideological views that are against like the counting of contributions. And they're just like, "I don't believe that that's something we should be doing. And so I want to remove myself from this list." Others are just like, "No, I don't want my username attached to that." And so there is actually a page on Wikipedia of people who have opted out of those kinds of it was more like people who have opted out of being in those like lists of the most active contributors. So we can also take a look at that. And that's something also whenever I do like a peer reviewed article, that's doing Wikipedia research, like I'l  always, like, check that list. If anyone shows up on that list, I'l tell them about it and say like, "Hey, this is a list of like opt outs that Wikipedians keep for, like not wanting to be publicly mentioned," even though officially, it's all public, and it's all creative commons license, and those people would admit that that you have the legal right to do that. But it's more of like, respect for person's kind of...</p>
  <p><b>Sara Mannheimer  </b>42:37</p>
  <p>Okay, sure. Okay, so did you go through that list and remove there anything that mentioned any, like, edits that they had made for those people? Or did you stil  include them in the dataset, but not? Like, talk about them in the paper?</p>
  <p><b>BSR02  </b>42:53</p>
  <p>Yeah, so we didn't talk about them in the paper, they're stil  in the dataset. We didn't remove them.</p>
  <p>Because there are objections on that is more of like being mentioned as part of an aggregate rather than necessarily... I don't know, like some people are more I don't think I'm realizing like some people are more ideologically opposed to the idea of like counting users, and like by edits in general, we didn't think I think that if we had been I don't know, it's actually a good question about whether we remove them from the data set. To be honest, I don't think it ever came up to us about the like, about whether we remove those people from the data set. I think that we kind of assumed that the real harm was in in mentioning them and mentioning their usernames. We didn't I mean, I think I think we also kind of felt like because we were doing a bit of a rebuttal study, I don't know. But now now, think about the fact that someone else does a harm doesn't mean that you could do that harm again. So I don't know. So I maybe we should have thought a little bit about like whether we should remove those. But it wasn't ever kind of articulated as like, "Get my data out of Wikipedia." I think there have been some people who have tried to like get their data deleted from Wikipedia. I'm not sure how I think about that. It's a good question. So I guess I guess I'm a little split maybe on, on removing people from the record, but I think</p>
  <p>it was more of like, we definitely didn't want to like even we didn't even reach out to anyone who was on that list. Yeah.</p>
  <p><b>Sara Mannheimer  </b>44:21</p>
  <p>Okay. Interesting. I want... do you know, what happened with those people who tried to get their data deleted from Wikipedia has? Have people been successful in that?</p>
  <p><b>BSR02  </b>44:31</p>
  <p>Only only for...Yeah. So no. And Wikipedia is like ideological y opposed, right to any... you know, it sees that as censorship. And often often it is people who have it's not that they're like contributors to Wikipedia who want their previous contributions removed, but they are, they are people who are depicted in Wikipedia, and they want their you know, whatever. Like, like, like a politician has something bad about them and they have hired some people to like scrub the Wikipedia article, but it keeps getting put back, because what could be like, the easiest way to get Wikipedians mad is to like do that. And so then they'l  they'l  like file a suit and Wikipedia gets sued, like all the time with these frivolous lawsuits, and they'l  just say, "Nope, section 230. Like we have the right," you know, unless unless there's like a copyright infringement, in which case, but like Wikipedians are super copyleft. And like, they're like, they they don't want to copyright material for their own kind of ideological reasons. So anything that's copyrighted can be taken down and deleted from the database. There are things called I can actually go into this for way too long about it. There is a process though, for removing personal information. So this is actually the oh, what's it cal ed? It's not super delete. I can find it later. But because sometimes some people get doxxed, where like, they're like, yeah, so some some it could be a public figure or some, you know, they they had privacy rights too or something, or like someone creates an article for someone who's not notable, but someone they're trying to harass, and they put, they put all their personal info and their phone number and their address. And so I don't know, social security number, I don't know, if you just do one of those things, and you're trying to harm someone.</p>
  <p>There are procedures for if, you know, especial y for if like nonpublic information is put on Wikipedia in that way. Those can be deleted from the database. And there are that is that has happened, it happens far too often. People upload nasty stuff to Wikipedia, people upload child porn to Wikipedia, people upload really nasty stuff. And there is you know, it's they do have all kinds of processes in place in the same way that like Facebook, and you know, all the major social media companies have those like il egal content filters that are available, and do those purging of things that are just like blatantly il egal content. But I would say like Wikipedia is, you know, very good at like removing things that are copyrighted, and things that are like il egal according to US law. And those do get purged from the database. But there is a record of the purging.</p>
  <p><b>Sara Mannheimer  </b>47:03</p>
  <p>Right, okay.</p>
  <p><b>BSR02  </b>47:06</p>
  <p>And the whole process, there's like a council that like, it's like one of the only secret parts of Wikipedia, but it's stil  like, its secrecy is documented I find, I've been meaning to write a paper about this. I find it fascinating.</p>
  <p><b>Sara Mannheimer  </b>47:17</p>
  <p>Hmm.</p>
  <p><b>BSR02  </b>47:18</p>
  <p>I mean, I probably won't, but I can give you pointers more if you're interested.</p>
  <p> <b>Sara Mannheimer  </b>47:23</p>
  <p>I think this is good. Okay, let's move to intellectual property, which we did discuss because basically everything on Wikipedia is CC licensed. Right. Okay.</p>
  <p><b>BSR02  </b>47:33</p>
  <p>That was done. It was easy.</p>
  <p><b>Sara Mannheimer  </b>47:35</p>
  <p>Yeah. Yeah, I kind of feel that is kind of all that we need to write here.</p>
  <p><b>BSR02  </b>47:41</p>
  <p>I mean, we also released all of our analysis code under—what did we do? I think we probably dual... I think we put a pretty permissive license on it. I think we dual licensed like CC and MIT for the code and documentation. Yeah, because we we don't see this as proprietary. You know, it's, it's, you know, we actually want this to be something that you know, because we were we were we were we were gonna fight. So we wanted, you know, as permissive as possible, remix this thing. We put a Binder up even so that people could I don't know if you know about MyBinder. Yes, we put up we like it works with Binder it works with, like, we're like "fork it, do whatever you want. Use it in your classes." Like no, we really wanted it to be like the spread, we wanted a permissive as possible copyright license. So.</p>
  <h2>48:32</h2>
  <p>But you succeeded, right? I mean, no one fought back? It was just those two little like...? Yeah, yeah.</p>
  <h2>48:38</h2>
  <p>Yeah, so some, some a couple of data science classes I know from some people have like, use this a bit as a, because we do it's like a good thing for students, because you can like walk through all of our Jupyter notebooks and our pipelines and stuff like that. So it's like a real world project. So yeah, it has gotten some uptake and some other educational settings. So yeah, we were we were happy about that.</p>
  <p><b>Sara Mannheimer  </b>49:01</p>
  <p>Really cool. Okay, are there any other issues or challenges that arose during this example that I haven't asked you about that you want to tell me?</p>
  <p><b>BSR02  </b>49:10</p>
  <p>Um, we did run into a bit of like, we want to, like the raw XML are actually not the raw but like, even the compressed like Gz, compressed XML dumps are 93 gigabytes, so that we wanted to actually save the version of it, you know, because, you know, we wanted to, for reproducibility purposes, want to get the exact same figures, and Wikipedia, like, deletes, dumps, like, they only keep dumps for a certain period of time, because they have to, like, you can't go get the now like October 2013 dumps from XML dumps because they just don't, they don't yeah, so we knew that they delete them after a period of time, but we wanted to keep our entire pipeline. So that was something we wanted to have, but then we tried to figure out where do we put those 93 gigabytes? It was like it's like one big I don't know, like, it was too big for Zenodo by default, and it was too big for Figshare by default. And so I think we actually contacted Zenodo. And we said, "Hey, like, we know that y'all are at CERN and do a bunch of stuff, can we get an exception?" We didn't hear back from them in the time period that we needed, I think it took them... And so we actually went to [a large university data repository]. So I emailed our data librarian, and our data librarian was like, "Yeah, no, you're not going to be able to upload this to the web interface. But, you know, let's, let's work with you." And our deal, everyone was really, really great in</p>
  <p>terms of, you know, at at [large university data repository], like opening up a back door to like, get us in to upload these, like hundred gigabytes of files of the 100 gigabyte file. And so that was, that was a little thing that I was like, "Oh, yeah, like, maybe I actually should’ve started with y'all at the beginning." So you know, so but we've got it there. And we have everything, the other legs Zenodo and Figshare.</p>
  <p>dumps have everything but that first one, and we actually redesigned our data analysis pipeline so that it pulls from [large university data repository] to get those if you if you run everything from from scratch at the beginning it no longer because Wikipedia has downloaded deleted those dump. And so now it pulls from  [large university data repository].</p>
  <p><b>Sara Mannheimer  </b>51:23</p>
  <p>Nice, okay, do you are you using like a transfer? Like a Globus sort of? Or do you just go little by little with like an FTP? Do you know?</p>
  <p><b>BSR02  </b>51:31</p>
  <p>No. No, this... No, it wasn't it was it was I think I just SSH'd a staging server and did it that way.</p>
  <p><b>Sara Mannheimer  </b>51:35</p>
  <p>I guess 93 gigs isn't that huge but...</p>
  <p><b>BSR02  </b>51:44</p>
  <p>Yeah, it's not like massive, but it was just like too big for the web interface. And so and then we were just using, we actually have, there is some there is like a cloud services kind of offering that Wikimedia has they use internally for all their stuff, and they make available to researchers. But we were we were going beyond the bounds of what that was useful for in terms of the storage. And then I actually ran a lot of these just on my I have like a beefy desktop. And so it was just running them locally for that and made sure. And then [my collaborator] also has like a beefy desktop. And so we were so we were like both running those and just synchronizing with get, and that worked.</p>
  <p><b>Sara Mannheimer  </b>52:27</p>
  <p>Okay, cool. Awesome. Oh, this has been so delightful. Thanks for talking to me.</p>
  <p> </p>


name: BSR_03
filename: Mannheimer_BSR03_Tra.pdf
created: 2024-06-21 19:17:44.511705
text_direction: LEFT_TO_RIGHT
contents:
  <p> BSR03_transcript_deidentified</p>

  <h2><b>SUMMARY KEYWORDS </b></h2>
  <p>data, users, book, author, data sets, records, link, recommendation, research, gender, published, context, ratings, public, people, set, scraping, project, integration <b>SPEAKERS </b></p>
  <p>Sara Mannheimer, BSR03</p>
  <p><b>Sara Mannheimer  </b>00:00</p>
  <p>Lest I forget, okay, there we go. So, first, can you tell me a bit more about yourself the type of research you do and the type of data you produce.</p>
  <p><b>BSR03  </b>00:15</p>
  <p>I am working on, so, recommender systems researcher is my primary background, with a human computer interaction focus to my training. But I do a lot of things particularly looking at issues of bias and discrimination, and search systems, recommender systems etc. I also do some other work on just making more effective recommendation research experiences. But with that we need a lot of data for....</p>
  <p>a recommender system is a personalized machine learning algorithm, we need data to train on to learn people's preferences. The recommender systems community has a long history of public datasets. Um, going back to well, originally, one of the earliest ones was the each movie data set that came from the movie rating and recommendation system operated by DEC in the 90s. And they made their data available to the research community. When they shut down, they transferred the data from their system to the GroupLens research lab at the University of Minnesota who used it to seed the MovieLens platform, which then produces the data sets that are probably used more than any other single data set for recommender systems. So it's that's a community that has—the current data set dumps have data from somewhere between 150 and 200,000 users who have used this thing over the years that they know they're participating in academic research. Minnesota provides dumps of the database for the recommender research community. Those kinds of data sets that need a lot of the core recommender systems work is needing data that we have, basically, we need to know what items users like whether it's movies, whether it's books, whatever. So I've done a lot of that with with doing that with movie data, particularly the movie lens data set. But then also working on research paper recommendation, which I mine Bibliography linking or link to bibliographies from the ACM Digital Library, they were giving me in-beta data dumps for a while. Um, and I've been doing some work on books where that data is, again, we need the data of what users have liked. And so there's an old data set that's been around for 15</p>
  <p>years or so from a website called BookCrossing. That predated Goodreads. I don't know how popular it was, I think it stil  exists. Um, but also, other groups have scraped and made available for academic research data from, from Amazon reviews and from Goodreads. And so those are the data sets I largely use. For the books and the kinds of data I use, there's the one of the tricky things is we don't actually care about the user identity. So it can be they can be deidentified, and I'm not going to go try to re identify them. There is research on redefining data users from these data sets. It's remarkably effective. But um, I just need to know that there is a user number 17. They liked these books. And so I can use that to train my recommendation models and and look at user behavior and look at the recommendation results and effectiveness.</p>
  <p> </p>
  <p><b>Sara Mannheimer  </b>04:03</p>
  <p>Cool, very interesting. Nice. Okay, so some of that is like social media data, others is established datasets that and it seems like you don't do any scraping or downloading yourself.</p>
  <p><b>BSR03  </b>04:20</p>
  <p>I haven't done any scraping myself in a long time. In grad school, I was doing scraping a little bit, but I had Yeah, I haven't done that in 15 years.</p>
  <p><b>Sara Mannheimer  </b>04:33</p>
  <p>Al  right. Um, so I think I sent you a follow up email asking you to identify a specific example of a recent time when you collected big social data or used social data to do research. The idea here is we want I want to be specific as we work through these issues. So having one project in mind is helpful. We can also sort of like, move around between projects, if there are some that apply more to the questions that we have in mind. But do you have a recent project that we might use as this specific example?</p>
  <p><b>BSR03  </b>05:16</p>
  <p>So probably the book data. So I've been with the, with some of my book research, particularly, I've been wanting to study. But generally, I've been wanting to study bias and discrimination and recommendation, particularly with regards to content provider attributes. So is the search engine biased towards or against content creators of particular ethnicities or genders? Um, getting that data is difficult.</p>
  <p>And I'm, and so I settled on books for two reasons. One is that, well, maybe one of the two reasons but books have, unlike, say, movies, books have an identifiable primary creator that we can think about, like there is and there might be two or three authors, but there's an author for a book. And while we probably want to be fair to everyone else involved in the book production process, too, you don't have the like movies, you want to be fair to directors, producers, actors, there's 17 of them. authors, authors are better defined than what we would want to do with say, movies. And then coupled with that the availability of rich data about books and authors. So that we can actually be able to identify authors, discrimination, relevant characteristics in many cases.</p>
  <p><b>Sara Mannheimer  </b>06:56</p>
  <p>Okay, so it's whether what is recommended is biased.</p>
  <p><b>BSR03  </b>07:02</p>
  <p>Yeah. I've also done some work on bias towards different types of users. But that's a much smaller off the shelf data project. This one... So this one, in order to answer the question, I tied together six different data sources. Um, I have book consumption records, BookCrossing, the Amazon reviews and Goodreads. And those were all scraped by others. Um, the Amazon and Goodreads data both came from Julian McCauley's group at University of California, San Diego. They have, they have a small cottage industry of scraping data sets and making them available. Um, and so that's where I'm getting the book consumption data to be able to train recommendation models and evaluate their effectiveness. But then I need book and author metadata. And so the Goodreads data comes with some book data. But the BookCrossing and Amazon data just comes with ISBN or ASIN. And so I have to go get that data from elsewhere. So I pulled in the, to get book metadata, I pulled in the Library of Congress, open MARC distribution records. And I pulled in Open Library. And then I linked... that has the book information, including who the authors are, but has no author information itself. Like, right, Open Library theoretical y does, but it's super, super sparse. Um, so I linked that with the virtual internet authority file to get author information. And that let me build a link of user rated book by author of gender.</p>
  <p> </p>
  <p><b>Sara Mannheimer  </b>09:00</p>
  <p>Nice, okay.</p>
  <p><b>BSR03  </b>09:01</p>
  <p>And then I could go do my study.</p>
  <p><b>Sara Mannheimer  </b>09:04</p>
  <p>Okay. Cool. That'l  be interesting when we talk about comparing data sets. Comparing and combining.</p>
  <p>Alright, so you've described your data collection method. Is this part of a grant funded project where you were required to treat the data in a certain way or if you had a data management plan?</p>
  <p><b>BSR03  </b>09:23</p>
  <p>Yes, the initial data collection predated the grant because it was used to drive preliminary results, but we're continuing to refine it through the grant process. And the data management plan, I specify that I'm going to share the data I can, okay, but note that some data is not going to be shareable either due to upstream restrictions—several of the data sets, I'm linking, I'm not allowed to redistribute. But almost anyone can go get the copy themselves, but I can't provide it. That's like the Amazon and Goodreads data is non redistributable. Um, I make sure I have a backup copy myself in case it ever gets taken down. But it's not redistributable or in other cases that haven't been triggered in this process, but my DMP stipulates that I can also not publish for human subjects protection or other ethical concerns.</p>
  <p><b>Sara Mannheimer  </b>10:19</p>
  <p>Okay.</p>
  <p><b>BSR03  </b>10:19</p>
  <p>So. But it also says that I have committed to making available the relevant code for reproducing my work, and where it comes from datasets that are public, but I can't redistribute. Basical y, I'l  give people the tools to go do what I did. And so all of the code for this data integration is open source. Anybody else that can get a copy can and has a half a terabyte of Postgres [PostgreSQL] sitting around can go to it.</p>
  <p><b>Sara Mannheimer  </b>10:46</p>
  <p>Okay. Who is your funder?</p>
  <p><b>BSR03  </b>10:49</p>
  <p>[U.S. federal funding agency].</p>
  <p><b>Sara Mannheimer  </b>10:50</p>
  <p>Okay, cool. Al  right. And so have you published any data so far? Or wil  that all happen when you're done with the project?</p>
  <p><b>BSR03  </b>11:00</p>
  <p>I haven't published any data, but we do have the codes available.</p>
  <p><b>Sara Mannheimer  </b>11:16</p>
  <p>Cool. Ah, all right, sweet. Al  right, let's move on to our core six questions. So the first one is about context. So I have a little blurb, I'm going to read you to help you understand how I'm thinking about context. "When we collect data from social media platforms, just as when we collected data in traditional spaces, context matters. But the context of a social media post may be absent or difficult to</p>
  <p>understand. These posts are by nature, short pieces of text taken from the larger context of personal and public life. And then this out of context effect is compounded when data are masked at a large scale." So that's kind of the issue that we're looking at here around context. Can you tell me about a time, if any, during the process of this research that you're doing when you considered the issue of maintaining and understanding this data's context. So like information about the community where the data was collected, or any contextual information about respondents or platforms.</p>
  <p><b>BSR03  </b>12:28</p>
  <p>So we haven't done a lot with context in this data set, the one thing we do do is that we don't mix and match consumption records. So we treat each of the consumption data sources as a separate rating data set, and we don't pull them together. So we'l  train a recommendation model on the Amazon review ratings, and use them to do recommendations for Amazon users wil  train a separate model on the Goodreads data and recommend for Goodreads users. So we're not treating user histories as fungible between sites. Um, that's about the only thing we're explicitly doing with context. Um, as I think about it, and context isn't something I have thought a lot about in the project. But our use of the data is very similar to contexts in which it is already being used, even if it's not the context in which the user intended it. So when you go to Goodreads, and you add books to your shelf, that's the records we get.</p>
  <p>We have that users added books to shelves, that they rate the book four stars, and we have timestamps associated with those. Um, that's the input Goodreads is using for their recommendation models. And so while the context the user is doing it in order to log their reading, or in order to communicate to their friends, what they're reading, in terms of the technology, like we're doing, with probably a different set of algorithms, exactly what Goodreads is doing to generate your set of recommendations. And so where we're effectively using a very similar context to try to understand how algorithms behave in that context in a way that I think is pretty faithful to the context of what's happening with, with the data in its original situation. It's at least on Goodreads, it's less clear on Amazon, because I don't know that Amazon is actually using ratings for any of the recommendations.</p>
  <p>We don't have users' purchase histories for very good reasons.</p>
  <p><b>Sara Mannheimer  </b>14:40</p>
  <p>Right. Yeah, I mean, that's part of what I am curious about too, is like, I guess since everything is deidentified in your situation, you don't have any knowledge of like, you know, this person, you know, from this region of the United States. These people tend to like these types of books or, you know...  users who identified this as their interests in their bios tend to like these types of books, or anything like that.</p>
  <p><b>BSR03  </b>15:07</p>
  <p>Yeah, if I wanted to go do some scraping of my own, I could get that. But because one of the things I also have that we haven't made much use of yet we haven't published anything on yet. We also have, but for the Amazon data and the Goodreads data, we have review text. And so one of my students is working on review based recommendation and making use of that review text. And they're, like, since it's review text, if I really wanted to figure out who a user is I and I could go figure out who wrote that review, because it's on, it's like, they just scraped Goodreads public reviews. Yeah, but I'm not going to go do that. Because re-identifying users is not the business for it.</p>
  <p><b>Sara Mannheimer  </b>15:53</p>
  <p>Yeah, um, but is that partially like, we have a question about privacy that we can go to... is that because you're concerned about privacy, or because you're concerned about liability, or just because that seems out of scope for you?</p>
  <p><b>BSR03  </b>16:10</p>
  <p>Um, partially out of scope, partially privacy, partially that as soon as I do that, I'm probably in personal information territory in terms of IRB oversight. And not that I've tried to skate around that in any way.</p>
  <p>But there's like, we're paying attention to the ethics of this work, but it, it changes, it changes what the nature of oversight document may change with the nature of oversight documentation would look like right now we have, we have to go ahead. But part of that's what the understanding of we don't have user identifiers.</p>
  <p><b>Sara Mannheimer  </b>16:52</p>
  <p>Mm hmm. Okay. Let's see. So I guess, so you just haven't thought too much about context, thinking that you're sort of using the data in the context in which it's supposed to be used almost. Okay. That that's helpful.</p>
  <p><b>BSR03  </b>17:21</p>
  <p>At least it's a similar context in which it is being used whether or not users think it's supposed to be used that way...</p>
  <p><b>Sara Mannheimer  </b>17:31</p>
  <p>Yeah. Okay.</p>
  <p><b>BSR03  </b>17:32</p>
  <p>Different people have different expectations.</p>
  <p><b>Sara Mannheimer  </b>17:34</p>
  <p>Right, okay.</p>
  <p><b>BSR03  </b>17:36</p>
  <p>But we're doing the same kind of thing with the data that Goodreads recommendation engineers wil  be doing except we don't have all of the data they have.</p>
  <p><b>Sara Mannheimer  </b>17:43</p>
  <p>Okay. Cool. Okay, let's move to data quality. Can you tell me about a time if any, during your example, when you consider the issue of data quality, so like, maybe like an incomplete population that you're drawing from or missing data of some type, or bots or bias? I know you're looking at that. But kind of thinking not like how you sort of dealt with data quality issues, and then also how you communicate any issues that you encounter to future people who might use your code or might try and reproduce your research.</p>
  <p><b>BSR03  </b>18:23</p>
  <p>So there are two sources of data quality issues with one of which we spend time thinking about. The first is the just the quality of the underlying source data. Which I don't spend too much time thinking about, at least on the rating data side, because, like for bots... Amazon and Goodreads are... their quality teams are working on kicking bots off the platform. They're going to do a far better job than I ever could. They have access to signals that aren't in the public data set that are what you actual y need to detect bots. Like there's work on detecting bots in rating data, but the rating data is missing all the signals like IP addresses, and timestamps and login activity and those things that like if I were actually trying to detect a bot, I'd look for who's publishing who's logging in from 15 IP addresses in one night, not who has an anomalous statistical pattern in what books they like. So I just "out of scope" that, I don't worry about it.</p>
  <p> </p>
  <p><b>Sara Mannheimer  </b>19:31</p>
  <p>Yeah.</p>
  <p><b>BSR03  </b>19:32</p>
  <p>Um, where I do start paying attention to more quality is on the metadata side. But particularly, we can also have quality issues with the data linking. And while we have—ostensibly have—a linking identifier, between the book rating record, or I mean the book consumption record, and the book metadata records to the ISBN, sometimes the ASIN. ISBNs are a holy mess. Um, we also have the issue of we want to do our recommendations at the work level instead of the individual edition level, just to be more realistic, and also to not have the data be near as sparse. So we have to link things together into works.</p>
  <p>We have a relatively decent algorithm for doing that with some very large known flaws we have to fix.</p>
  <p>Um, about 10% of the Goodreads ratings are for this supercluster work that contains Tom Sawyer, Treasure Island, Little Women, one other famous children's book, and then like a dozen other lesser, well known books.</p>
  <p><b>Sara Mannheimer  </b>20:50</p>
  <p>Oh, wow.</p>
  <p><b>BSR03  </b>20:53</p>
  <p>But the reason for that is the reason for that is relatively straightforward. There was a joint edition of all four of those books published with a single ISBN. And so we're using ISBN, and book records and a clustering algorithm to identify what's probably the same work. And we don't yet have—other than manually playing whack a mole with known bad ISBNs—we don't have a good way to go and break those clusters that are being inappropriately linked. That's one known data quality problem. We just say it's there and hand wave it.</p>
  <p><b>Sara Mannheimer  </b>21:27</p>
  <p>And when you say it's there, is that like in the place where your data is available? in your manuscript, in both?</p>
  <p><b>BSR03  </b>21:32</p>
  <p>I think we document it on the data page. I do not remember, I don't think we... do we say? I guess I don't specifically state the extent of the problem on that page, I just say there are some known problems that can cause books to be inappropriately linked. Um, we try to be transparent about that; we haven't clearly documented the like—"Whoa, there's this one mega cluster."</p>
  <p><b>Sara Mannheimer  </b>22:03</p>
  <p>Yeah. Would you do that? Have you published on this yet? Like, can you do that in your methods, or like...</p>
  <p><b>BSR03  </b>22:10</p>
  <p>What we do... So we roll it into some other statistics, because in our data integration... So we do this linking, to link a rating to its cluster. And then we try to identify the author gender for a cluster. And the way we do that is we take the first author from every book record associated with the cluster, we take them now there's no linking identifiers. In the book records in the virtual internet authority file, we have to do name matching. And so we but we go with the premise that it is very unlikely for two authors to have the same full name and different genders. And so it may happen, but we treat it as very unlikely.</p>
  <p>So we take the first author from every book in the cluster, we take every virtual internet authority file, record, in our, um, we take every that matches that one of one of those names, we take their gender, if</p>
  <p>they have one, and then we do a collapse. So that we basical y, it becomes since virtual enter an authority file strips out non binary gender identities, a problem we document, working on fixing it, that's another quality issue we pay attention to is like, is it doesn't have gender, right? Um, in all cases, but we've been we've resulted in is is an as a cluster is a clusters, author, gender is male, female, or ambiguous or unknown. And so what that manifests, we also have like, what do we do if the user rates multiple has multiple ratings for the same cluster? That doesn't happen very often. An acceptably small fra-and we document the fraction acceptably small fraction of users have multiple ratings for the same cluster. Um, and then we have, so this manifests as okay, users are more likely to have multiple ratings for it, and users are, and it also has an ambiguous gender.</p>
  <p><b>Sara Mannheimer  </b>24:27</p>
  <p>Yeah.</p>
  <p><b>BSR03  </b>24:28</p>
  <p>Um, and so we report statistics on ambiguous author gender and on multiple ratings for the same author for the for the same cluster. This is just the biggest one of that problem.</p>
  <p><b>Sara Mannheimer  </b>24:42</p>
  <p>Yeah. So I guess part of what you're doing to support this quality issue is just having more data. So then it becomes an exceptionally small fraction, right? Is that what you're saying?</p>
  <p><b>BSR03  </b>24:55</p>
  <p>Partially and then also, like, I want to fix that supercluster linking problem I just haven't found the solution. And yet, some of it so being and we're gonna keep updating the data set codes that people can, can go and, and try to. So people can get the current version, I don't know if I have a full log of known problems anywhere, but I also have mental list of like, okay, here are the open problems that I need to try to fix with this data. So the next project uses a better version of it.</p>
  <p><b>Sara Mannheimer  </b>25:27</p>
  <p>Nice.</p>
  <p><b>BSR03  </b>25:29</p>
  <p>And we try to we try to approach it, we approach it and we're explicit about this in the paper to learn what we can from the data we have while being transparent and clear about its limitations.</p>
  <p><b>Sara Mannheimer  </b>25:42</p>
  <p>Yeah. Okay. Great. Cool. Let's move to number three data comparability. So during this project, did you compare and combine and or combine multiple big social data sets? And you say, yes, yes. And so why did you combine them? How did they advance your research? And then what strategies and challenges did you encounter and how did you address address those?</p>
  <p><b>BSR03  </b>26:13</p>
  <p>So we have described a chunk of the, of the combination process, and it's described in more detail both on the documentation page and on. And in our journal paper, that was published in user modeling and user adaptive interaction earlier this year. Um, but we, the reason we needed to was two reasons. Um, so we needed to combine multiple data sources, because no one data source had all of the data we needed. We had ratings, but we didn't know who wrote the book. We know who wrote the book, but we didn't know anything about them as a person. And so we needed to link different things together in order to be able to actually answer that question. Also, then we have these three different sources of of</p>
  <p>user book interaction data. And that's partially because the recommender systems research community, likes seeing results on multiple data sets.</p>
  <p><b>Sara Mannheimer  </b>27:13</p>
  <p>Okay.</p>
  <p><b>BSR03  </b>27:15</p>
  <p>Um, but also, it gives us the BookCrossing data set and the Amazon data sets have been used in a lot of recommendation projects like yours, you'l  see lots of papers using both of them. BookCrossing, a little less these days, but a lot of older book recommendation work used it. And so that gives us this historical continuity and comparability. But then Goodreads is by far the largest book recommend data set station data set I know of, it also gives us in addition to the rating, so Amazon, we only have user ratings, Goodreads gives us add to shelf actions. So anytime we have a log of users adding books to their public shelves, and so that lets us detect expression like that. They added it to a shelf with some kind of expression of interest. They read it, lets us get that interaction data, even if the user hasn't gone to express a writing. Okay, and most production recommendation systems are trained on interaction data, not on reading data. And so it lets us that one was we have historical continuity with the first two data sets, and then Goodreads lets us get a lot closer to actual practice.</p>
  <p><b>Sara Mannheimer  </b>28:32</p>
  <p>And then what challenges I guess you've talked about some of them with, like imperfect metadata, and also imperfect data in each of the data sets, but what other challenges did you encounter? And then what strategies did you use to address them when you were working with all these datasets together?</p>
  <p><b>BSR03  </b>28:52</p>
  <p>Um, so some of it was one of them was just document well, efficiently integrating them and documenting that integration. So when we go to write it up, it's easy to tell what the heck we actually did.</p>
  <p><b>Sara Mannheimer  </b>29:04</p>
  <p>Yeah.</p>
  <p><b>BSR03  </b>29:05</p>
  <p>And so the way we approached that, in the version that's currently public, is we did all of the data integration in PostgreSQL. And so we imported the data set in as close to raw form as possible, um, directly in the Postgres tables, and then our different cleaning and integration, extraction, cleaning and integration steps, with a few exceptions were written as SQL queries. And so if I needed to go write down exactly how we did the author name matching, I can just go look at one SQL query and see the exact code that I'm, I'm currently working on retooling that a bit to not depend on Postgres. But the core concept is relatively the same that each discrete step has is a place I can go and see what I did there.</p>
  <p>In order to have a clear documentation on how the data was linked together.</p>
  <p><b>Sara Mannheimer  </b>30:08</p>
  <p>Yeah.</p>
  <p><b>BSR03  </b>30:08</p>
  <p>We also go ahead.</p>
  <p><b>Sara Mannheimer  </b>30:10</p>
  <p>That's included in the public data that you provide as well, so that anyone can go match it. Yeah. Okay, cool.</p>
  <p><b>BSR03  </b>30:18</p>
  <p>Um, we also have a, I had some challenges with just the size of the data set my recommendation software, some of the algorithms choked on the size of some of the data. So some of it, we just weren't able to use some of it, we I went to improve the software performance, we could actually run on the Goodreads data. Um, those were a lot of the big issues. We did see, I believe gotten the paper, we've got some charts that are, say, comparing gender distributions between different data sets. Um, and there were there were differences, say in the proportion of of authors of books by female authors rated in the different data sets, not huge differences, but but the Goodreads data was more user engagement book presents was more equitably distributed than the Amazon review data, for example. The Amazon review data is also older, we're working with the 2014 data set at the time. Okay, but we saw some differences there. But that's about the extent or you we compare it on a basic statistical level, how big they were, what the sparsity was. But beyond that we didn't. We didn't do a lot, particularly we haven't done any qualitative comparison of what the difference is in these data sets, these data are.</p>
  <p><b>Sara Mannheimer  </b>31:45</p>
  <p>Okay. Al  right. Awesome. So let's move on to informed consent. Was there a time during the process this research when you considered informed consent for the users themselves?</p>
  <p><b>BSR03  </b>32:04</p>
  <p>So other than the nagging thought, in the back of my mind that we kind of hand waved that? Not a, it has not been an explicit thing, so we're only working with public records. And there there is a I mean, there is a contextual integrity thing here of that, like the when the user submitted the right review to Goodreads, using it in my research wasn't their intention. Um, but there is a but we have we are working entirely with public records. Um, this is standard practice for recommender systems research.</p>
  <p>Um, there's, there's good arguments that perhaps it shouldn't be, but it is standard practice. Um, that's about the extent of the thinking there. We thought about other of the ethical issues like we have it this paper or another paper we have that ethical note in.... I'm just searching here to see if it was this one.</p>
  <p>Okay, it's not this one. I put a note, in another paper that was actually working with user demographic data, that that because the data sets are public, like the work we did does not expose users to any risks to which they were not already exposed. So we're, we're trying to be careful that we're not exposing users to new risks, but, or authors. Right. But um, the informed we have not like, yeah, informed consent from the users participate, or from the users whose data we're using.</p>
  <p><b>Sara Mannheimer  </b>34:13</p>
  <p>Yeah, and this fine answer too, it's like, you're saying that in your case, it wasn't something that you considered that you needed? I think that's fine as well, because that's part of what's so complicated here is like the line of like, when it is human subjects data, and when it's not, is blurry. So I think this answer is totally helpful, too.</p>
  <p><b>BSR03  </b>34:38</p>
  <p>And the recent relevant research communities are having, having discussions about like, what should we actually be doing with this data or not doing with it and there are problems with oh, it's, it's all public, so I just used it. Um, I think that our use is significantly less ethically fraught than some of those. It's just public. So I just used it projects. Right? Um, but there's a there's the need to pay ongoing attention to that, for that discussion.</p>
  <p> </p>
  <p><b>Sara Mannheimer  </b>35:13</p>
  <p>And I guess I do I like who, you know, when you're paying attention to this clearly you haven't aren't knowledgeable about it, but like, who do you read? Or like, what sources? do you go to? What if you were to have like a conundrum? You know, if it was, if you didn't feel it was as straightforward as this research, like, what would be your sort of strategy there?</p>
  <p><b>BSR03  </b>35:33</p>
  <p>So I people I've read... Casey Fiesler, and then I've also I kind of keep a loose eye on the general work of the PERVADE team. Um, Anna Lauren Hoffman.... who I would go to... I may go to one of them, or just generally, my friends, particularly my colleagues in the fairness, accountability, and transparency community. Several there I could, would ask, if I or and also having a dialogue with our local IRB folks.</p>
  <p>And so, um, why while this particular  project has been, they've they've judged it not human subjects research, because no personally identifiable data, and we're not engaging with users. Yeah, um, we've talked with them about other projects about this and related projects, and the ethical dimensions have been beyond just the human subjects concern. And they've been a useful resource. So I also talked with them.</p>
  <p><b>Sara Mannheimer  </b>36:38</p>
  <p>Cool. Okay, that's helpful. Al  right, let's go to number five out of six, privacy and confidentiality. So we kind of talked about this a little like, I guess it was there a time when you consider the issue of privacy?</p>
  <p>I guess, since it's all deidentified, I think you're or maybe if not, in this example, in other examples that you've worked on. So I'm kind of like trying to get into like times when you had issues and that you thought about.</p>
  <p><b>BSR03  </b>37:11</p>
  <p>So this data, like there's the de-identifiability portion, but I think more importantly, there's the it's al public portion.</p>
  <p><b>Sara Mannheimer  </b>37:20</p>
  <p>Yeah.</p>
  <p><b>BSR03  </b>37:21</p>
  <p>Because if someone wanted to use this data, to one of its subject's detriment. It's not anything I'm doing that's going to enable that.</p>
  <p><b>Sara Mannheimer  </b>37:34</p>
  <p>Right. Yep. Okay.</p>
  <p><b>BSR03  </b>37:36</p>
  <p>Um, and so there's not like, what I'm doing doesn't create any new privacy related harms may create other harms, in some cases, but it's not creating any new privacy related harms for anyone. Because the data so out there, yeah, and I'm not redistributing any of it. But, um, there is a so we've been doing some other if we if we're in discussions that we haven't acted on yet, of trying to move beyond data, or beyond book authors who are relatively public individuals in terms of you get an authority record. And we're using the authority records to say academic scholars or others who don't have as large profile.</p>
  <p>Um, how do we get the data? And what do we do once we have it? And even if it's even if it's assembled from public sources, say we have if we have [Mechanical] Turkers, getting data from scholars' public profiles, which we have not done, and I don't know that we're actually going to, but if you assemble it in one place, it makes it easier to find. And so we would not make that data publicly</p>
  <p>available. sources are public, anyone can go do what we did.  But nope, not gonna make that publicly available. Um, we're also looking at TREC, the Text REtrieval Conference, and their fair ranking track and privacy concerns and putting data in one place concerns along with identity just the the concerns of people's ability to control their presentation of their own identities, is one of the reasons why we didn't use something like gender for the Academic Search data sets that we put together for that. Um, instead, we used the economic divide level of the country in which a researcher was operating when they wrote a paper as a crude proxy for the level of resources they likely had available for research.</p>
  <p>The idea being that we want to be fair towards schol-, we want to make sure that scholars who are operating and under resourced environments have a fair opportunity to have their research work surfaced in search results. And that's something that's derivable from public information, the affiliation country in the paper you published because it was all published papers. And then combined with public records like the International Monetary fund's economic classifications, or there's so that we tried to go with something that is completely derivable from public data and does not involve any sensitive personal attributes. So we could catalyze this kind of research without creating new privacy or discrimination problems through making a an archival data set available.</p>
  <p><b>Sara Mannheimer  </b>39:19</p>
  <p>Right.  Oh, the so that, that's very interesting. So you, like actually designed your research in a way to like, after thinking about privacy issues, like, okay. Very interesting. Okay. Cool. Al  right, let's move to intellectual property. Was there a time during your research when you considered intellectual property concerns, either by from the platform or the participants users themselves? Did you consult with anything, anyone about the concerns? What strategies did you use to sort of figure it out and address any challenges that you encountered?</p>
  <p><b>BSR03  </b>41:54</p>
  <p>Um, I haven't, I haven't spent significant time or effort on that, um, that like, from a lot from an intellectual property liability perspective, the people who scraped and initially produced the data would be on the hook. Yeah, that's one reason I'm not redistributing the data. Um, and on its usability, like the data sets are very well known and are stil  available.</p>
  <p><b>Sara Mannheimer  </b>42:22</p>
  <p>Okay, yeah. So you're kind of assuming since they've been public for so long, there aren't any big issues? Yeah.</p>
  <p><b>BSR03  </b>42:29</p>
  <p>Or at least I mean, and other datasets have been taken down. So if Goodreads or Amazon wanted him to start distributing these data sets they would have asked him to? And so...</p>
  <p><b>Sara Mannheimer  </b>42:42</p>
  <p>Did you... So in other projects, has this come up for you as an issue, like looking at Terms of Service, or you know, trying to figure out any issues you might encounter?</p>
  <p><b>BSR03  </b>42:52</p>
  <p>It's one of the reasons I've been hesitant to do a bunch of scraping myself is just to avoid that set of issues. Outsourced it, I've done scraping of sites that have explicit open access terms. But, um, but I mostly outsource that.</p>
  <p><b>Sara Mannheimer  </b>43:12</p>
  <p>Okay. Al  right. Great, well, are there any additional issues or challenges that arose during your example that I haven't asked you about?</p>
  <p><b>BSR03  </b>43:24</p>
  <p>Um, so one of the, I brought it up briefly, one of the complex ones, like especially if we're dealing with author gender is what is that and how was it recorded, it is recorded accurately.</p>
  <p><b>Sara Mannheimer  </b>43:38</p>
  <p>Yeah.</p>
  <p><b>BSR03  </b>43:39</p>
  <p>And we like the when you look at the market authority file specification for the gender field (375, I think), um, it specifies that it is gender identity, specifical y, it specifies that it is, um, it can come from an open vocabulary. And there can be multiple gender identity records with starting in validity dates. So like the file format itself, has this very rich notion of how it can record an author's gender identity, and it's specific that we're talking about gender identity. Um, you throw all of that into the woodchipper. A virtual internet authority file, and the nuance disappears. Um, but so we paid attention to what the data is and what it means where it comes from, to the extent that we can. And then where we know our current integration has significant problems in this front. We're looking at how to get better author demographic data particularly we can get it from the Library of Congress name authority file, instead of going through VIIF, their data is higher quality. The coverage is at least in our initial prototype integration, that we're stil  debugging, the coverage is very, very poor. So thinking through that set of issues also, but and also being being encouraged by the thought that was put in, in both in the mark record design, and then also the program for cooperative cataloging things working group report on Recommended Practices for recording author, identity, author, gender identity, in a name, authority record, um, is very, very thoughtful. And I believe anybody feeding into the Library of Congress files, probably following that. But VIIF sorts sources for many national libraries, many of which GCC is the thing. And so they have their own standards for how the how the the records are created and maintained. And so like, seeing the dealing with a very nuanced and complex subject, seeing pieces of the data ecosystem, handling it in ways that seemed relatively good. And then that gets fed through a data integration that just kind of chops it up.</p>
  <p><b>Sara Mannheimer  </b>46:29</p>
  <p>Yeah.</p>
  <p><b>BSR03  </b>46:30</p>
  <p>Um, the fact that we are looking only at statistical properties, and not looking at individual author identities, except for debugging purposes, means that the manifestations of the problem are different than they would be if we were, say, publishing a list of here's the genders of these authors. Um, but we stil  have... one of the things that's unknown, and I would like to get a handle on, is the extent to which accuracy and presence of these labels is label-dependent. How does the distribution of the data we don't have or the data that's wrong, compared to the distribution of what we do have? Because if authors of particular gender identities are systematically more likely to have their identities erroneously recorded, which I think is almost certainly true, that then does skew the statistical results. Open question is, what's the magnitude of those skews? I have a rough sense for what the direction probably is, but what's the magnitude? Um, we also see, and sometimes there's evidence in the data that the direction might not always be what we expect, like I've looked at the proportion of books written—the male to female ratio, in Library of Congress over time by publication date since about 1960-ish. And it's flat, which surprised me.</p>
  <p> <b>Sara Mannheimer  </b>48:20</p>
  <p>So the number of women authors has not gone up?</p>
  <p><b>BSR03  </b>48:24</p>
  <p>The fraction.</p>
  <p><b>Sara Mannheimer  </b>48:26</p>
  <p>Yeah, the ratio. Hmm. Interesting.</p>
  <p><b>BSR03  </b>48:29</p>
  <p>So that made me wonder, has the fraction really not gone up? Or is there a bias in which authors people pay attention to creating a complete and accurate authority record for? That then skews our perception of the space? And I don't know the answer to that.</p>
  <p><b>Sara Mannheimer  </b>48:52</p>
  <p>Oh, there's so many factors. Wow, very interesting.</p>
  <p><b>BSR03  </b>48:58</p>
  <p>I would like an like an answer to that. But I don't have an answer to that. There's a lot of open questions in the data. But those are some of the things that we're thinking about too, is like, how do we, how do we deal with this? How do we deal with a sensitive and very personal demographic attribute in a way that's respectful? And we try to pay attention to that. Not going to claim that we do it perfectly. But we're trying to pay attention to that and do the best we can with the data we have.</p>
  <p><b>Sara Mannheimer  </b>49:29</p>
  <p>Yeah. Okay, I think that's all that I have. I do have one favor to ask is, I'l  send you a follow up email after this. Just saying thanks. And I wonder if you could, if you have anyone in mind, who also does big social research, or qualitative research, if you know, people who do that, or who publish their qualitative data. I'm looking for additional people to interview, who might be interested in this and be wil ing to spend an hour talking to me, so. Okay. Thank you so much. This was really great. It was so interesting to hear about all of these challenges. And I think it'l  be really helpful to my data set. So thanks again.</p>
  <p>Al  right, have a great day.</p>
  <p><b>BSR03  </b>50:24</p>
  <p>You too.</p>
  <p><b>Sara Mannheimer  </b>50:25</p>
  <p>Okay, bye.</p>
  <p><b>BSR03  </b>50:26</p>
  <p>Bye.</p>
  <p> </p>


name: BSR_05
filename: Mannheimer_BSR05_Tra.pdf
created: 2024-06-21 19:18:02.905674
text_direction: LEFT_TO_RIGHT
contents:
  <p> BSR05_transcript_deidentified</p>

  <h2><b>SUMMARY KEYWORDS </b></h2>
  <p>tweets, data, twitter, people, collected, api, research, conversations, question, big, paper, irb, set, users, context, weighing, disclosures, sexual violence</p>

  <h2><b>SPEAKERS </b></h2>
  <p>Sara Mannheimer, BSR05</p>
  <p><b>Sara Mannheimer  </b>00:04</p>
  <p>Okay, so my name is Sara. I'm a data librarian at Montana State University. And I'm also getting my PhD from Humboldt University in Berlin. And I'm working on this research that the idea is that big social data (like social media posts, and blogs, and photos and videos) and archived qualitative data are similar because they're both presented online for reuse or and when you when you do research, you reuse them. And so I am trying to connect these two communities of practice through their data curation practices. So the idea is that ultimately, we data librarians and repository managers wil  be able to support this research, more research support the responsible creation of this research, and maybe data sharing as well. So I have conducted a lit review that were identified these six key issues that are present in both big social data and qualitative data reuse. And so the interview is structured around those six issues. So I have like an introductory question six questions about your practices. And then one wrap up question. So it should take about an hour, maybe 75 minutes, depending on how much we talk. Any questions before we begin?</p>
  <p><b>BSR05  </b>01:34</p>
  <p>Uh, no I think it makes sense.</p>
  <p><b>Sara Mannheimer  </b>01:36</p>
  <p>Okay, cool. So tell me about you the type of research you do, and then the type of data that you either collect or produce.</p>
  <p><b>BSR05  </b>01:44</p>
  <p>Yep. Right. I'm a PhD candidate at [university, program]. I focus on online communication networks, and a lot of online political communication. So things related to hashtag activism and polarization and misinformation and all that fun stuff. And so a lot of my, or, you know, almost all the data I work with is some kind of social media data, basically. And most of what I have done, and currently do is with Twitter data specifically, I've done a lot with Twitter data. I've touched other social media data like Reddit, I've been playing a little bit with TikTok. I've looked at the documentation for all sorts of other ones, you know, Facebook, and YouTube and things like that. But most of what I do is Twitter. And so what that looks like is a lot of the times I've worked with data that was collected by someone else at our institution. So someone else who is running something like a decahose feed, where it's just 10%, of all tweets, or something like that. And so I've worked with that. I've also worked with, you know, data that I pulled myself by keywords or something like that, or pulling user timelines directly from the API. I think I've rehydrated tweets. So I've used other people's tweet IDs to get those myself. And then there's another group I work with maintains a panel of Twitter users where they continuously collect timelines for an effort. So different forms of Twitter data.</p>
  <p> <b>Sara Mannheimer  </b>03:49</p>
  <p>Great, perfect. So I asked you to identify one specific example, when you were collecting data or reusing data that was collected by someone else or preparing your own data for publication or sharing.</p>
  <p>Can you tell me about a recent example? And we can go beyond the example but it's helpful to have something specific as we talk through the the issues.</p>
  <p><b>BSR05  </b>04:16</p>
  <p>Yeah. So we did a project a few years ago, one or two years ago on Me Too. And so this was all Twitter data. And so what that looked like was—I actually met someone who is a friend and colleague of mine now who was at a different institution, and their institution had access to like, all tweets somehow, like they paid like a really big subscription to be able to have really full access. And at one point he had collected the Me Too tweets. So, for the period that was kind of immediately following when it first started taking off, so he had something like the first three months or something like that. And he told me about this, and he was able to share the tweet IDs with me. And so I rehydrated that data. And we ended up using that data for a project on looking at how people disclosed early in the hashtag campaign, and how that may have produced stigma around kind of disclosing for other women to disclose experiencing sexual violence. So it was kind of a sensitive data set. And so we worked with... a kind of standard, just keyword data set, so it was like tweets with Me Too or something like that. And we want to go publish it. I like the idea of, you know, trying to share data, it seems like an important data set, because it is Me Too. But I was concerned because, you know, because it is like a data set about sexual violence, basically. And a big part of our paper was we came up with a way of identifying which tweets were disclosures which were not, and we have to describe the method in the paper. And more or less, that's a method to identify survivors of sexual violence in our in our data set. And so we we talked about this among ourselves. The team I was on, which was my advisor, and then another PhD Student and [their] advisor. And what we ended up doing was, we were aware, you know, we can just put the tweet IDs out there. That's fine, with Twitter terms of service. But what we did was we ended up hosting the data through [data repository], and we were able to get kind of a—they wil  do like a restricted use agreement that people have to sign to do that. And so I think that involves proof of IRB, or exemption, and they have to sign like that they they won't kind of misuse it and stuff like that. So that kind of barrier to using the tweets felt like a fairly strong deterrent, or strong enough, given the work, you stil  have to do after that to do anything bad with it.</p>
  <p><b>Sara Mannheimer  </b>07:58</p>
  <p>So you've published just the tweet IDs under restricted access in [data repository]. Nice.</p>
  <p><b>BSR05  </b>08:03</p>
  <p>Yeah. So that was, that was kind of the full story of the data that we worked with. There's a portion in there where we also bought just a little bit of missing data from Twitter.</p>
  <p><b>Sara Mannheimer  </b>08:18</p>
  <p>So you used.. or, your col aborator used the Twitter API to pull down tweets with the Me Too hashtag and then you've got some additional data. Okay. And was the project grant funded? Like, so, did you have like a data management plan or anything that you needed to follow?</p>
  <p><b>BSR05  </b>08:39</p>
  <p>Um, I think it's funded by an internal grant, but no one kind of told me I had to follow any particular guidelines with it. So it mostly just had it on our lab computer, which, at the time only I had access to, and that I had set that up to be as secure as I could, but yeah.</p>
  <p> <b>Sara Mannheimer  </b>09:06</p>
  <p>And let's see, Do you have plans for storing, retaining and deleting the data in the future?</p>
  <p><b>BSR05  </b>09:15</p>
  <p>I stil  have the Me Too data somewhere. I think it's on backed up hard drives at this point. It's actually not on the lab computer anymore, I think because I had to reset the whole computer. But I do have a backup of the tweets themselves somewhere. I probably wasn't planning on deleting them anytime soon, because it's a useful data set that we come back to so I've actually used that data set again in a later project. A case study. Yeah, the, the tweet IDs are kind of posted for perpetuity.</p>
  <p><b>Sara Mannheimer  </b>09:56</p>
  <p>Perfect. Okay. So we'l  start with the six issues which are context, data quality, data comparability, informed consent, privacy, and intel ectual property. And so with context, I have a little quote to help you to help communicate what I mean by this. So Halavais suggests that when we collect data from social media platforms, just as when we collect data in traditional spaces, context matters. But the context of a social media post can be absent or difficult to understand, because posts are by nature, short pieces of text, images, videos, etc, that are taken from the larger context of a personal and public life. And then the out of context effect can be compounded when data are amassed at a large scale. So coming from this idea of context, can you tell me about a time, if any, during the process of your research with the Me Too Twitter data, when you considered the issue of maintaining and understanding the data's context? So like context about the community where the data was collected, or context about the Twitter users themselves?</p>
  <p><b>BSR05  </b>11:13</p>
  <p>Yeah. Yeah. So we did a mixed method study. And so what that looked like was, we did a big inductive qualitative analysis where we identified themes in the Me Too tweets. But a big part of that was your like, the broadest way you can sort the tweets for those that are disclosures and those that are not. So there's a whole set of those tweets that are disclosures, and so we read 1000—1000-plus tweets that were—</p>
  <p><b>Sara Mannheimer  </b>11:47</p>
  <p>Oh, just manually, okay.</p>
  <p><b>BSR05  </b>11:49</p>
  <p>Yeah, yeah. So maybe we sampled 2500, or something like, is about half of them that were disclosures in the end, and is about 60% of those that, like were stories of some sort, rather than just the hashtag Me Too. And so I, I was glad we read those, because you really got a sense for what happened at that moment, and who is talking. But there was there were some tweets, sometimes where you, like, to understand you had to go... Like you couldn't tell just by the tweet text, like you had to go see, like, who was saying this? Some of them are a little ambiguous, like, are they mocking Me Too? Are they serious? Like, what is it? Is this person disclosing or is this person just making a general comment?</p>
  <p>And sometimes it's hard because sometimes it was profiles to more anonymous, or they weren't established figures or anything like that. And we come back to this question sometimes, like who those people are in terms of what communities they're coming from, because you didn't have you don't have demographic data for those folks. So it's kind of a, almost like a sleight of hand that we do in the paper, the whole paper is framed around women, but we never actually like, check who was a woman or not, it's just kind of going off the assumption that most people are disclosing more women. And so that's something we kind of think about sometimes is the fact that we don't, you know, we don't know what</p>
  <p>communities were underrepresented either in those disclosures. And, yeah, and then, I guess, just maybe this is more technical, like the context of like, when you're like doing qualitative analysis, and you're reading the tweet. And it's just like a text box in front of you versus like seeing the tweet with the picture in it. And, like the username and the person, like, there's a lot of times when it didn't become clear until you went to the tweet itself, versus just like in a spreadsheet or something like that.</p>
  <p><b>Sara Mannheimer  </b>13:59</p>
  <p>Right. And when you were publishing the data in [data repository], did you do anything—like did you address this in the description that you put there? Or is most of the way you address this through the paper?</p>
  <p><b>BSR05  </b>14:17</p>
  <p>Yeah, I think in terms of... so we did with our, with the Tweet IDs, like we didn't really say anything about like, which we'd identified as disclosures, anything like that. So in that sense, that context, isn't there. The only context so it's really there in the description is just, you know, what kinds of tweets they were and those were the ones containing Me Too the what period they cover. And, you know, anything we knew about any other kind of those like details of how we sample, but nothing really more about those communities that they might be coming from.</p>
  <p><b>Sara Mannheimer  </b>14:59</p>
  <p>Um, Do you think that your research was affected by the sort of fuzzy context that happened here? And then what strategies did you use to sort of make up for that in your research?</p>
  <p><b>BSR05  </b>15:23</p>
  <p>Yeah, I think I think just generally, doing the mixed methods, where the qualitative proceeded the quant[itative] really helped us understand the data better. And I could see the project having gone down like the wrong route without the qualitative analysis, because it really gave us a sense of what was going on in the data, because we had to look at so much of it.</p>
  <p><b>Sara Mannheimer  </b>15:52</p>
  <p>What was the quantitative analysis part? What did you do next?</p>
  <p><b>BSR05  </b>15:56</p>
  <p>So the quant part was when we, so we had the tweets, labeled as disclosures and not disclosures. So we used that as a training set to build a predictive classifier for the rest of the tweets. And then most of the rest was a network analysis.</p>
  <p><b>Sara Mannheimer  </b>16:15</p>
  <p>Okay. Cool. Al  right. I think that's good. Let's move to data quality. So can you tell me a time, if any, during the process of this research, when you considered the issue of data quality? So you mentioned, like not knowing whether the tweet was making fun of Me Too or actually disclosing? That's one example. Or bots or another example, or missing data? I guess you did talk about buying some missing data from Twitter, too. But are there other examples when you took that into account?</p>
  <p><b>BSR05  </b>16:55</p>
  <p>Yeah, there's actually so when my friend originally collected the data... so, Me Too was October 2017.</p>
  <p>So I think [my friend] col ected it in February or March 2018. So already, that was a few months. And then I met [my friend] the summer of 2018. And then I don't think I rehydrated the tweets until December of 2018. So by the time I rehydrated them might have been over a year since Me Too first</p>
  <p>happened. And so even comparing against [my friend's] tweet IDs, so I had all the tweet IDs and then I had hydrated some of them. I think we were missing about 25% of them. And it's because either they had been taken down or probably more likely they'd been deleted. And so I always kind of wonder what those tweets were. Because some of our results... Those tweets if they were taken down for certain reasons, it's almost like our results could understate like, the parts of me too, that weren't positive. So it could have been folks who didn't have a good experience from disclosing online or it could have been, like, for example, you didn't find a lot of... surprisingly, we didn't find that many tweets that were, like harassing people or disparaging the hashtag, even though that's common now. But it could have been because those have been, were more likely to be taken down or something like that. So that data missingness is something that was kind of always in the background, and there was just nothing we could really do about it. So that's for the major thing when I think of data missingness. So you had mentioned... remind me of some of the other things you mentioned?</p>
  <p><b>Sara Mannheimer  </b>16:57</p>
  <p>Um, bots... and</p>
  <p><b>BSR05  </b>18:56</p>
  <p>Oh bots, yeah, yeah. When we were going through like, you notice there's some there, luckily, like it didn't seem like there's too much. But there's like some wonky stuff, like, you find spam kind of data.</p>
  <p>And it, you know, has the hashtag in it. But it's not related to Me Too. We did some things with the follower graphs of the people in our data. And that's always a little tough to work with, because the Twitter API rate limits on those are so strict that you can't always get as much of those networks as you want. So yeah. Oh, and then the missing period. Yeah, my friend had happened to miss something like a 12 hour period when [they were] collecting data. So the tweets, the tweet IDs weren't even in the data set from [my friend]. Yeah, we didn't think it would affect the final analysis and it really didn't, but it's kind of... I only noticed it because I went down to doing like a time series that was at a particular resolution, like if you kept it at a certain resolution you couldn't even tell. And so I could have easily missed it.</p>
  <p><b>Sara Mannheimer  </b>20:19</p>
  <p>And so let's see, what strategies did you use them to, like communicate or clarify these issues to, like in your paper or to future users?</p>
  <p><b>BSR05  </b>20:37</p>
  <p>In the paper, just trying to like be straightforward about it was seemed like the best bet, like 25% of tweet IDs—like that is a fair bit. On the other hand, there's so many papers out there that are made with like, 10% of all tweets. So like, it's a lot better than those at least. So, so yeah, we, like being straightforward about communicating that there. And then with the, like, kind of the description of the data online, kind of the same thing, just trying to be as clear as possible, how the data was collected.</p>
  <p>And then like, I think most people who work with Twitter data regularly know this, but kind of emphasizing that, you know, this data is collected at this time. That was like a year after Me Too. And so what this includes is Me Too tweets, but not those have been deleted by the time of data collection.</p>
  <p>So</p>
  <p><b>Sara Mannheimer  </b>21:36</p>
  <p>Yeah. Did you work with a curator at [the data repository] to like, help work through how to just like, include these descriptions with the data? Or did you deposit just on your own?</p>
  <p><b>BSR05  </b>21:48</p>
  <p>I think I wrote up a lot of it. And then I think it was [curator name]? at [data repository], who helps kind of walk us through the rest of the process?</p>
  <p><b>Sara Mannheimer  </b>22:01</p>
  <p>Yeah. Great. Alright, let's go to data comparability. So during your example, did you compare or combine multiple big social data sets?</p>
  <p><b>BSR05  </b>22:14</p>
  <p>For that project, we didn't really. There's like the tweet data we had, and then we bought some, but it's like the same data. Yeah, this one was pretty contained. I'd say. If I were to, like, give a different example, just for this. There's this... one of the groups I work with, they've maintained this panel data.</p>
  <p>And what that is, is they have public voter registration files, so, voter registration is public in the United States. So they have those. So they're real people, what they did was they match those to Twitter accounts. On Twitter. So what that does is it brings in the demographic information with the Twitter account, so you can start to ask questions about like, what are real people doing on Twitter versus this weird mix of real people and bots and organizations and stuff like that. So that's probably the biggest instance of the data that I work with that's like, combined with a different data set.</p>
  <p><b>Sara Mannheimer  </b>22:48</p>
  <p>What demographic info is in the voter registration files, like gender? And...</p>
  <p><b>BSR05  </b>23:37</p>
  <p>Yeah, the ones that come directly with it are gender, age and state of residence, some states report race, but there's other fields that are inferred by... it's a it's a commercial vendor who kind of assembles the voter file because they come from all these different sources. So they assemble it and then they add additional metadata that they infer based on proprietary classifiers and stuff like that. So we we've done some work here to like validate the proprietary stuff, and it's not too bad. But they that's how we have inferences. And sometimes it's different. Sometimes they add information for race and sometimes for political party if it's not reported.</p>
  <p><b>Sara Mannheimer  </b>24:32</p>
  <p>Did you encounter any challenges when you were trying to combine the Twitter data and the voter registration files?</p>
  <p><b>BSR05  </b>24:40</p>
  <p>I didn't do the matching myself, but I'm pretty aware of the process and yeah so just, you know, matching names is a difficult thing because of informalities and stuff like that, multiple people having the same name and same location, and I think they ended up just doing... the most basic thing they did was if there were multiple people in the same location, they just said they couldn't match the file. And so it's really a panel of people with unique names and unique location.</p>
  <p><b>Sara Mannheimer  </b>25:18</p>
  <p>And who used their actual name as their Twitter handle, right. Or as their descriptor. Okay.</p>
  <p><b>BSR05  </b>25:23</p>
  <p>Yeah. So they actually have to use their full name, either in their handle or their username, which there's a lot of people who don't do that. And that project was... that's a separate project, so that one was IRB approved, because I know, it can be a little fuzzy doing that kind of linkage.</p>
  <p> </p>
  <p><b>Sara Mannheimer  </b>25:43</p>
  <p>Oh, okay. So the previous the Me Too data, did you go through IRB there or consult with them?</p>
  <p><b>BSR05  </b>25:54</p>
  <p>Yeah, my my advisor generally has a kind of IRB approval for working with Twitter data and online activism, because that's [my advisor's] much broader gig. I don't think we had to submit a separate one to work with me to data. Yeah.</p>
  <p><b>Sara Mannheimer  </b>26:13</p>
  <p>Do you know anything about that? Because I know, like different IRBs have different levels of the way that they sort of see Twitter data, you know, like, at MSU, they just, it's, like, exempt? Because it's existing data.</p>
  <p><b>BSR05  </b>26:26</p>
  <p>Yeah, I have seen the IRB... I don't think it was one that went through as straight exempt. And I think it's partly because in the IRB, they also talked about how they would, how they would or would not, quote, individual tweets and things like that. So there's a lot of language. And I saw, I saw one of like, the rough drafts, and about how they were going to, like, make decisions about that based on... and if they would kind of reach out to people about getting their consent to include their quotes or something like that.</p>
  <p><b>Sara Mannheimer  </b>27:06</p>
  <p>Okay. Well, that's a great segue. Our next issue is informed consent. So can you tell me about a time if there were any during the Me Too example, when you considered informed consent?</p>
  <p><b>BSR05  </b>27:28</p>
  <p>So let me think just for a moment. It didn't come up directly at all in terms of like working with the tweets, so we didn't discuss it for that. I know the big thing that we talked about it with was in terms of kind of using tweets as examples or quotes or something like that. So the general guideline in my lab, that my advisor has us following, that I agree with, is we generally try not to quote people who aren't like public, big public figures or who, like, wouldn't expect that their tweet could be quoted, so not like the tweet that got two retweets versus the one that got 100,000 or something like that. But most of the disclosures were not like popular tweets, they were pretty small tweets. So what we ended up doing for those was altering, we didn't report actually direct quotes, we altered the text. And we do like altered texts, and we mismatch—mash together, like similar tweets, so that, hopefully, they shouldn't be identifiable. Like, you shouldn't be able to reverse look them up or something like that. And I know there's kind of ethics there to the thing, reporting the data.</p>
  <p><b>Sara Mannheimer  </b>28:57</p>
  <p>Yeah. So in the paper, you said, an altered example of a, you know, tweet.. is this.</p>
  <p><b>BSR05  </b>29:09</p>
  <p>Yeah, exactly. Oh, and I don't know if I actually say anything about informed consent. There are instances or sometimes like a tweet is, like, a real y good example. And it might be that one we use...</p>
  <p>this didn't come up in that paper. But that is a case where we would, like, reach out via Twitter, like via Twitter DM to ask someone. And I know that's something... my advisor has a book on hashtag activism, and for like the big examples that they have, they reached out to people, since they knew they were going to be kind of memorialized in the book.</p>
  <p> </p>
  <p><b>Sara Mannheimer  </b>29:44</p>
  <p>Yeah. Um, have you or have you ever consulted with anyone about like, practices around informed consent? How have you come to these strategies?</p>
  <p><b>BSR05  </b>29:57</p>
  <p>Yeah, so most, a lot of strategies come through my advisor, who has been working with Twitter data and this kind of stuff for a long time, and I noticed [they], a lot of [their[ practices come from, you know, working with [their] collaborators and kind of trying on, like black feminist practices and things like that in the literature. And then, you know, I've taken classes where research ethics are part of the conversation. And I've done my own kind of reading in general, on research ethics and thinking about these kinds of things with Twitter data.</p>
  <p><b>Sara Mannheimer  </b>30:40</p>
  <p>Yeah, have you found that it's more common when you're looking at research ethics literature to find a discussion of this? Because I feel like, you know, with IRBs not having caught up, there's not like a lot of official guidance. But yeah.</p>
  <p><b>BSR05  </b>30:56</p>
  <p>Yeah. This is like one of my little hil s that I've been trying to die on, is like to bring those statements a little more forward in papers. So you tend to either tend to not see them? Or you tend to see them as like an, oh, by the way, like, we got it approved by the IRB, or like, oh, by the way, like, this is maybe an ethical issue, but we're at the end of the paper, so like, there's that so. So I like, I tend to not see it with my papers, I've been trying to almost put it towards the front of the Methods section, because ethics should come before the methods really ideal y, but. This is my hil  to die on, because my co-authors always are trying to like make it shorter.</p>
  <p><b>Sara Mannheimer  </b>31:46</p>
  <p>But do you feel that—I guess it depends from project to project. But do you have like a sense that participants expect to give consent for a project like this, like a hashtag project?</p>
  <p><b>BSR05  </b>32:04</p>
  <p>My understanding... there's been like some research on this. Is that most Twitter users don't always know that their data is being used like this. My understanding is most are okay with it being—they would be okay with it being used if it was like being used for good ends... like whatever those are. But so like, for Me, Too, though, like I could very easily understand someone being perturbed about being in that data set. And so that's... So it's we kind of try to do it as a balance. Do we think this research is important enough? And given how, if we think it is important enough, you know, what safeguards can we put in place to make sure that this person isn't going to face harm from being in the data set? So the hosting in [a social-science specific data repository] versus just like [a general data repository] or something like that was one of those safeguards that we put up?</p>
  <p><b>Sara Mannheimer  </b>33:08</p>
  <p>Interesting. Awesome. Al  right. Let's move to—Let's move to privacy and confidentiality. So can you tell me a time during the research when you considered issues of privacy? Like restricting access to your tweet IDs is one example. Or protecting the data during the research, or thinking about the expectations of the users? The Twitter users. Yeah.</p>
  <p><b>BSR05  </b>33:40</p>
  <p>So the big one is like, we don't want people to be able to easily identify survivors of sexual violence.</p>
  <p>And that was, we, I think we had several conversations about that among ourselves, trying to figure out the best way to—like if we could share the data responsibly. Because I think if we hadn't identified [the data repository we used] as an option, we wouldn't have shared the tweet IDs. We thought that'd be a little too far. Especially given that like our paper, it's not easy. like it'd be almost easier, just to like manually look up the hashtag Me Too, and harass people through there, than, like, use our tweet IDs, build a classifier after having read your own, like 1000s of tweets. It's not really practical that anyone would actually misuse our method. But it's stil  like it describes the way of identifying a certain group of sexual violence survivors. And so that was kind of a conversation for us in terms of privacy. There's like smaller thoughts I had about... like no one like, really was watching me with the data and so I was always trying to wonder like, how many copies of this data should I have? I had like one.. I basically had one of my laptop and I had one on the lab computer. And like, where should these be. And it's like, now I don't have the copy on my laptop. But because the more copies there are like, the more chances there are that someone is gonna be able to touch it that shouldn't be able to touch it. And at the time, it didn't seem like a big deal, because I was the only one in the lab. And I was like, the only one working with the data. But over the past year or so, like I've started to bring more people onto the lab and onto the computer. And I have to make sure you know. I love all the people in my lab, and I trust them. But I'm not always going to be the one managing the computer in the lab. So.</p>
  <p><b>Sara Mannheimer  </b>34:28</p>
  <p>Yeah. Tell me more about the conversations you had with your colleagues, and like how you worked out these ideas about privacy and how you would handle it.</p>
  <p><b>BSR05  </b>35:58</p>
  <p>Yeah, there was, I would say... There's a nice kind of range of like how people felt about sharing data.</p>
  <p>So there was one professor who felt that sharing the Tweet IDs is like the right thing to do, it's good to be kind of open with this. Especially since you can share Tweet IDs. And kind of on the other end, there was one of the other professors who was much more hesitant. And so [they] pump the brakes on a lot of those conversations, or [they try] to raise concerns about it. Because—I'm trying to remember, there's like this specific concern. Oh, it's because, I forgot we did some web scraping for this too. I'm sorry, maybe... I don't know if it's relevant. But I feel like I should mention this too, because it's just a another example. So at the time that we collected the data, you, you couldn't get replies to tweets through the Twitter API. So it's like only the keyword tweets. But we've felt that it was really important to have replies to Me Too tweets, because that's the conversational part. And so we're part of like, what we're looking at is like, how survivors might talk to each other or support each other. We have plans for future projects around that. But you can't get those through the Twitter API. And so it kind of put us into position where the only way to get them was to do the web scraping, which is against Twitter Terms of Service, technically. So I collect... I don't have major qualms with like, bending Twitter's Terms of Service with web scraping. Like I wasn't like mean to their servers, like I put in pauses, so I wasn't like hammering them or something like that. And I got the replies, to Me Too tweets, I get the reply threads.</p>
  <p>But we had conversations, that came up about whether we should use them and whether... not like whether we should tell people we use them, but like, yeah, whether we should use them in the analysis, which would affect whether like we report it in the paper or not. Because to report it in the paper is basically to say like officially, like we broke the terms of service, and you have to justify breaking the terms of service, because... and that's, you know, because the terms of service aren't like ethical rules.</p>
  <p>They're just a set of guidelines set by a corporate company to like, protect themselves. So the users...</p>
  <p>so we had conversations about that, too, which kind of... There's one advisor who was like, I do not care what Twitter thinks like, let's just use the tweets. And another who's like, like really does very understandably did did not want to, like, get in trouble over this or something like that. So we kind of</p>
  <p>talked back and forth over that, trying to decide. So like, these kind of back and forth conversations where a lot of... I guess like just walking around the ethical markers, like trying to weigh, like, this is the benefit of doing this, versus this is the cost. So with the scraping, it's like the benefit is—you get this much fuller picture of me too. Versus there's potentially legal consequences here. And then with the releasing of tweet IDs, you know, the benefits of open science, being able to facilitate further research on this. Versus the cost, which is like potential harm to the participants. And so I think what pushed that conversation into releasing them was that I was able to propose [the data repository we used] as kind of this—I think someone referred to it like as a walled garden approach, like the data is there, you can see it, but like, there's a wall around it, that only certain people can get through. So I don't know if that answered that last question that you were looking for, but...</p>
  <p><b>Sara Mannheimer  </b>40:57</p>
  <p>Definitely, yeah, this is really interesting, because you talked at the beginning about sort of weighing whether the research was important enough to have some, you know, some danger to the participants—some risk. And then you talk about, like, weighing openness and privacy. So I'm seeing like a lot of ideas at odds, where you're weighing is the research important? Is openness, important enough? Is following those terms of service important? So it's interesting to hear how these conversations went with your group. And so with the... this idea of the web scraping, I feel like this is going into my last question about intellectual property. Like, how did you resolve that conflict with the web scraping benefit and costs?</p>
  <p><b>BSR05  </b>41:55</p>
  <p>Um actually, I know, like, something pushed us toward a final decision, but I can't remember exactly what it was. I can't remember. I'm not sure what it was. But I so I think part of that conversation, though, is like, yeah, if you could get the tweets via the API, you could, you could just do that. And so there's part of the question is like, why can't you get them through the API. And at the time, and I stil  believe this was probably because... it was probably just like, technically annoying for them to like, set up the API so you could get replies. And so the reason you couldn't get replies wasn't because they were like trying to protect users in some kind of way. And that is kind of one of my major beliefs is, Twitter isn't really doing this to protect users. If they're doing it, they're doing it to protect themselves.</p>
  <p>Computationally or something like that. And so looking at it as the users' tweets, they're just the same as all the other tweets that we can get. It's just a matter of how we got them. And you know, I understand like Twitter... it's an international website, like they cannot have 1000s of web crawlers hitting their website all the time. But so you know, the little thing I told you is just, you know, be nice about going through it, like not making any more requests than like a very fast user could make or something like that.</p>
  <p><b>Sara Mannheimer  </b>42:47</p>
  <p>And so then did you end up talking about it in your paper, too, that you had to use the web scraping and said that you felt that it was small enough that it wasn't against the Terms of Service?</p>
  <p><b>BSR05  </b>43:48</p>
  <p>Yeah, we ended up... yeah, I think, um, I'l  send you the paper after this so that you have the reference for it. I know, we mention it. So we say... and it's kind of, we don't explicitly say web scraping, I think, because what we say is more like we used the package to do it. So we just say, we used this package to get replies. But like, here, anyone who works with Twitter data, you're gonna see that package and know that that's not an API package. I'm not... I'm sure... I think I wanted to include a discussion of it.</p>
  <p>But it got struck out because of the concerns. The potential legal concerns. I remember I wrote a discussion at one point, and I think it got struck from the draft.</p>
  <p> <b>Sara Mannheimer  </b>44:45</p>
  <p>Interesting. And so like, Well, I think that's enough, actually. Yeah. This is a common... You know, I've heard data curators talk about breaking the terms of service at times, and I feel like as academics, we're like, we just have different values and priorities than a corporation. And so it's interesting, like, if I'm following my values, which is to get the full data and create a research project that's meaningful, like, Is it okay to bend a little bit?</p>
  <p><b>BSR05  </b>45:24</p>
  <p>Yeah, I think I've heard someone say, if you're like a researcher, especial y like PhD student, one of the questions you have to ask, and I think is a valid question, is like, what if you did scrape it against the terms of service? And then like, you couldn't publish your PhD because they came after you? And that's like, a scary thought. I don't think it's ever happened to anyone. But I think that's the kind of risk it does pose? Yeah.</p>
  <p><b>Sara Mannheimer  </b>45:57</p>
  <p>Al  right, great. So that's all of my like structured questions. Are there any issues or challenges that arose during the research that I had didn't ask you about? Or that you expected me to ask? And I didn't ask?</p>
  <p><b>BSR05  </b>46:15</p>
  <p>No, not too much. I think I always think of the panel data that I work with is the most... I mean, the survivor data is obviously ethically tricky. But the panel that I stil  actively work with the most ethically tricky. Just because it's matching datasets. So I think we're always trying to balance that one. So like, only using it for questions that meet that level of importance of using the panel? Because I know we get questions about it. But I think one thing... maybe this is just a general comment. When it comes to like matching data sets, or working with, like, these highly sensitive data sets, like sexual violence survivors, is, um, I wish sometimes these conversations were a little more like the weighing stuff. Whereas I feel like it's really easy sometimes to just say, like, that's risky, and so they shouldn't do it. And then there's the other end, it's like, they don't think about it, they just do it. I don't, I think I think, in working with big data, there needs to be a little more room for thinking about, like, this is risky data, but we stil  can do something with it in the same way that, you know, biologists work with risky chemicals, or not even biology, but like, you know, medical records are very sensitive. But like we have them, we use them.</p>
  <p>And there's reasons for using them. But there's then like certain precautions go around those. So I think yeah, that's just something to think about sometimes is, how can we work with this risky data, or data that might be risky? And make sure that we're treating it the proper way? So that it's like acceptable to be taking that level of risk?</p>
  <p><b>Sara Mannheimer  </b>48:10</p>
  <p>Yeah, that's part of my research. I feel in a way. It's like, how do we... Yeah, how do we make this happen? And I'm hoping like, through the interviews, I'l  be able to find people on both sides of the spectrum, like very cautious people, and people who are like, eh, whatever. And then we'l  see. We'l see from that, like, if there are ways that data curators can sort of help people weigh those options and help see their side. So yeah, cool. Well, this has been amazing. It's so fun to hear about your project and all of the challenges that you encountered. Thanks for taking the time.</p>
  <p><b>BSR05  </b>50:07</p>
  <p>Yeah, I hope it's helpful.</p>
  <p> </p>


name: BSR_06
filename: Mannheimer_BSR06_Tra.pdf
created: 2024-07-11 14:33:35.299520
text_direction: LEFT_TO_RIGHT
contents:
  <p> BSR06_transcript_deidentified</p>

  <h2><b>SUMMARY KEYWORDS </b></h2>
  <p>tweets, data, twitter, people, gathered, publish, hurricane, disaster, community, specific, paper, privacy, felt, experiencing, understanding, algorithm, fingerprint, text, events <b>SPEAKERS </b></p>
  <p>Sara Mannheimer, BSRO6</p>
  <p><b>Sara Mannheimer  </b>00:00</p>
  <p>Right. Um, okay, so tel  me about the type of research that you do and the type of data that you produce or use.</p>
  <p><b>BSRO6  </b>00:09</p>
  <p>Yeah. So specifical y around like big social data, social media data, my work is focused on understanding how communities respond to natural disasters and climate change. And we do this by, by gathering large amounts of Twitter data specifically, it's sort of before, during, and after large scale crisis events, either doing it in real time, sort of like as an event is going on, or going through and trying to retroactively pull the sort of Twitter information in specific geographic areas or related to specific keywords. And then a lot of what I do is designing algorithms are sort of like data science methods to parse through this data, and try to correlate or in some way predict how communities are going to respond to these disasters, using the sort of discussions that we see on social media. So a lot of it is aiming to get to a more socially derived understanding of a resilient community. So it's not just like, how many power outages did you have it's, can we learn something about the dynamics of the community, based on what people are saying on social media in an effort to try to predict or to try to improve this for, for the future. We can figure what makes this community resilient, maybe I can bottle that and give it to this community to make them resilient to something else, or the same disaster, etc.</p>
  <p><b>Sara Mannheimer  </b>01:28</p>
  <p>Hmm. Oh, that's so cool. Al  right. Nice. Can you please describe, so then, in my email to you originally, I'd ask you to come up with a specific project that you've done recently, that we can use as an example.</p>
  <p>Okay. We don't have to stick to the project the whole time. But it's nice to have some specifics when we're going through these issues. Yeah. So tell me about the example that you selected.</p>
  <p><b>BSRO6  </b>01:57</p>
  <p>Cool. Yeah. So this is one specific paper that.... So again, at the idea, this, this paper was sort of like motivated by a particular like algorithm and data processing, you know, series of steps that I developed through my PhD. And it's, the idea is that a community, the aim is to try to parse out, there's a lot of, you can get a lot of information out of Twitter, there's a lot of very, like, not rich and a lot of very rich data. And so we're trying to look for just additional ways to gather information from these from just a large body of tweets. And so this was, this data processing algorithm, this umm... it's called resilience fingerprints, to like just to make this like terminology easier, is derived from a more social scientific understanding of what makes a community resilient. So sociological ideas of how people might communicate with one another or relate to one another. And essentially, we break down this concept of resilience into five sort of categories. And this is like social ties to one another, maybe more</p>
  <p>institutional, but stil  social ties, things like churches, or like aid groups, things, ways that people interact in a community that are in a more structured way. There's like, of course, like institutions, police, fire hospitals, there's physical infrastructure, there's, there's roads, there's right the things that we built in our environment. And then there is health and like quality of life, so the actual healthcare outcomes that people are experiencing. And essentially, what we we do in this algorithm is, is look for, for keywords or sort of terms related to these different categories, and how they relate or they show up together. So like one example is someone tweets, 'I have run out of power, relying on neighbors for flashlights and batteries.' That's got an infrastructure component and the like, power is out. And it's got a social component, and like I'm relying on my neighbor, so we're, we're looking for those sorts of ties. And essentially, what this does is those ties we become, I'm looking to see how often does this category in this category line up with one another. And we call that a fingerprint. And so in this study, we then calculated these fingerprints for the 15 different sort of like large crisis events. These are things like hurricanes, these are earthquakes, a couple political events, like the shootings in Charlottesvil e a few years ago, or there was some Irish political unrest when they tried to put a constitutional ban on abortions. We looked at the sort of Twitter discourse around that, and the Twitter discourse around Brexit, and we calculated these fingerprints to try to tie these resilience components to specific attributes or specific elements of the disasters themselves. So we see things like in a hurricane, people talk a lot about infrastructure. In an earthquake, people talk a lot about infrastructure, but not quite as much. In a hurricane, people talk more about social ties than in an earthquake which is more political, and looking for these sorts of relationships to try to then understand, like what are the things I should be paying attention to, in a particular disaster. So that's sort of like... the flavor of it is trying to use the social media data in this rich, interpersonal textual communication that's happening online, to inform a better understanding of what parts of my community are being stressed or are being utilized during some sort of a crisis.</p>
  <p><b>Sara Mannheimer  </b>05:23</p>
  <p>And you're in engineering, right? And so, in engineering, like, do you, how intensive is the social science training? Or is it more... Is it like a mixture of both social and sort of hard science training?</p>
  <p><b>BSRO6  </b>05:38</p>
  <p>It is, there is not a lot of social science training. This is something that specifical y we reached out to collaborators from a communications department, or communications background to help do a better job of understanding, to help us understand how to process information or to have a--I'm trying to remember the, the terminology, make sure I'm being correct with this-- to develop a cognitive processing model, which describes, the aim was to say someone experiences an event, then they tweet something, we wanted to have a sense of how they got from an experience to a tweet. And so we interacted with and used a lot of expertise from some people in communications to try to have a better sense of it. As an engineer, I would just, that's something that would totally get washed away. And so we really wanted to make sure that, that was grounded in a better sort of like communication or sociological theory.</p>
  <p><b>Sara Mannheimer  </b>06:35</p>
  <p>Okay, so you use a collaboration to do that?</p>
  <p><b>BSRO6  </b>06:38</p>
  <p>Yes, yes, exactly.</p>
  <p><b>Sara Mannheimer</b> 06:40</p>
  <p>Perfect. Cool. Um, what was your data collection method? Did you use an, the Twitter API?</p>
  <p> <b>BSRO6  </b>06:46</p>
  <p>So we, I used hydrated... so I used a bunch of existing tweet datasets. So I was able to basically just scrape anything I could find off of the internet, someone had pre-curated a list of tweet IDs, and then did the process of turning those IDs back into the original tweet content. So all of these datasets were pre-gathered by other people. And that gave us a larger temporal horizon. So we can look at events in different parts of the world, we can look at events over a longer period of time, rather than trying to like query things in real time.</p>
  <p><b>Sara Mannheimer  </b>07:19</p>
  <p>Okay, great. So you reuse other people's social media data?</p>
  <p><b>BSRO6  </b>07:24</p>
  <p>Yeah, exactly. Al , and I think all of it. Yes. Al  of it was reused.</p>
  <p><b>Sara Mannheimer  </b>07:28</p>
  <p>How do you find that? Would you look for projects that had done similar studies? Or...?</p>
  <p><b>BSRO6  </b>07:33</p>
  <p>Yeah, so a lot of it was from, it was easy enough to find like a specific....Hurricane Sandy was a hurricane where there was a bunch of, and I'm sure you're familiar with this, Twitter changed their policies to having like before, I think 2014, your coor-your physical GPS coordinates were attached to every tweet by default. In 2014, it became an opt in thing. So it's sort of like neutered that as like a, as a research data source. Hurricane Sandy was like the last major natural disaster to have occurred before that. So in this, like, algorithms for processing Twitter space, there's a ton of studies that do this on Hurricane Sandy, because you have GPS coordinates for every tweet location through Hurricane Sandy. So this was like a starting point. There's like one paper specifically that is cited as like the definitive Hurricane Sandy data source. And so we were able to look through like downstream citations of that to see like, who else had cited that or what did they cite? And that started to like, branch out into a couple other different repositories, but it was all looking through other papers. And then people just fortunately, publishing their data along with it or maybe sent them an email to try to get a data set. So everything that we were using was previously attached to another paper in some way.</p>
  <p><b>Sara Mannheimer  </b>08:51</p>
  <p>And there's like some construction going on outside. Are you hearing background noise or?</p>
  <p><b>BSRO6  </b>08:56</p>
  <p>No, not at all. Good mic.</p>
  <p><b>Sara Mannheimer  </b>08:59</p>
  <p>Okay, good. Yeah, they're redoing the loading dock and they're like pounding concrete. Great. Al  right.</p>
  <p>Was this grant funded, like a situation where you had a data management plan or some other like requirements for how you handled data?</p>
  <p><b>BSRO6  </b>09:15</p>
  <p>There were no requirements for how we handled this. This was like a sort of a pilot study that was being done to provide some preliminary results that were going to go into a grant. But all of this was, was handled just... the restrictions, the only restrictions we had on like data handling and management</p>
  <p>were governed by Twitter's private policy and the the things you sign up for when you have a Twitter developer account or do any of those things.</p>
  <p><b>Sara Mannheimer  </b>09:40</p>
  <p>Yeah. Okay. And then, did you publish data from your example? Or did you just point to your sources?</p>
  <p><b>BSRO6  </b>09:48</p>
  <p>I have them all compiled and it is linked in the paper, but specifical y requested that any any of, the term I think it was like any use of these other datasets should be directed at their appropriate sources, right?</p>
  <p>Like I put them all here for convenience. But if you're going to do anything with this, cite—notionally, right, if you're going to do anything with this, cite the original ones don't cite this one as the source for that. Here's the CSV file, but cite the other, the other people when you, if you want to use this down the line.</p>
  <p><b>Sara Mannheimer  </b>10:16</p>
  <p>Nice, okay. And did you publish in a repository or as supplementary materials with your paper?</p>
  <p><b>BSRO6  </b>10:23</p>
  <p>Yeah, it's a, it's in like, this was done at [university]. And we have like, the libraries have a data repository where you can put a DOI on it, and easily share it. So it's, it's called the [institutional data repository name]. And it was a pretty easy step to do to just hold this data sort of in addition to the paper.</p>
  <p><b>Sara Mannheimer  </b>10:41</p>
  <p>Perfect. Okay, great. So we'l  get started with these six issues. Yeah. The first one is context. And I have like a little paragraph to help describe what I mean by context. So when we collect social data from social media platforms, just like in traditional spaces, when we used to collect data context is important. However, the context of a social media post could be absent or difficult to understand. Social media posts are short pieces of text or little images or videos that are taken out of the larger context of somebody's public and personal life. And then the out of context effect can be compounded when data are collected at a large scale. So this is kind of the issue that I'm thinking about here. Can you tell me about a time if there were any during your research when you consider the issue of maintaining or understanding the Twitter, the tweets context? So like contextual information about the community where you collected the data or information about individual respondents?</p>
  <p><b>BSRO6  </b>11:51</p>
  <p>Yeah, absolutely. And so I can think of two specific examples that I can touch on. One is a difference between like, languages, the, some of the default Twitter settings for, for just like scraping data from if you're using certain software to just scrape Twitter data, defaults to English tweets. And in somewhere like New York, that's, that leaves out a lot of people, in somewhere like Miami, that leaves out a lot of people. So this was something we were very cognizant of, was not necessarily...We ended up just doing sort of like systematic double checks to if I went back and then queried for the same period of time, how many tweets in this geographic area, how many tweets are English versus non-English, and making sure that our samples that we gathered, sort of, were not if the tweets in this area were 90%, non English, and we were using a 90% English data set? Right? There's there's a disconnect there. So one was looking at the context of the place with specifical y focusing on language. And the other this is something we noticed pretty early is that there's a big difference between pulling one tweet from a person who has maybe said the key word Hurricane Sandy. And so they've ended up in a data set.</p>
  <p>What we ended up doing for this was not for this particular study that I'm talking about, this is another</p>
  <p>one. So this is moving away from that original project. But this is a time this has happened. Is, is it turns out to work a lot better, we have way more complete, or way more representative data to find the people and then go try to retrieve their series of tweets. So I might be a person that's tweeted 100 times during Hurricane Sandy, but only once did I say the word Hurricane Sandy. And that would be the only tweet that would show up in a Hurricane Sandy database. So what we found to work a lot better was to look for the users who would show up in this database and try to pull as many of their tweets to that time period as possible. Exactly to provide the context and not just get their one out of context tweet that might show up in a data set.</p>
  <p><b>Sara Mannheimer  </b>13:56</p>
  <p>So did you collect additional data in, in addition to what you had pulled from other...?</p>
  <p><b>BSRO6  </b>14:02</p>
  <p>Yeah, so that was this was again, for a, for a fol ow up study. But yeah, we did that exactly was to take the original set of tweets, which you know, may be collected via various means, but they're all somehow related to a disaster. And if a person should, instead of looking at the data as a measure of content, if a person showed up in there, then we would go and actually pull all of their additional data so we could see those people over time.</p>
  <p><b>Sara Mannheimer  </b>14:43</p>
  <p>Okay. So let's see. Did you consult with anybody or consider other research projects or for the literature like how did you come up with, how did you identify these issues? And then how did you come up with your strategies for alleviating the issues?</p>
  <p><b>BSRO6  </b>15:03</p>
  <p>So I think maybe it's a little bit like a like a lucky guess the, the kind of, in this studying this idea of community resilience is incredibly place specific. And that's like if you're going to do an engineer— you know, a very technical engineering study of the resilience of a community, it is all contextualized by the place, the place specificity is incredibly important. So this was something that was like we knew was going if we're studying community resilience, because that is so largely discussed in this field of community resilience, that that was the reason why we went to this is something that needed to be addressed. Truth be told, if that was not something that was like, present in this research domain to begin with, I'm not sure how that would have, like, come up. There, there wasn't like a specific set of guidelines that we were looking at, or something they would say, you know, and you need to check the language, because that's, that's an important context. It was very much that, you know, I knew if I'm studying community resilience, I need to have the context. And so in this domain, that was important.</p>
  <p><b>Sara Mannheimer  </b>16:04</p>
  <p>Yeah. So you were using, like, knowledge that you had gathered through other studies? And from your just knowledge of the domain?</p>
  <p><b>BSRO6  </b>16:10</p>
  <p>Yeah, yeah, sort of existing work in the literature, which says, look, the way that this community responds wil  be different than this, you have to consider that community level that place specificity that that context is important.</p>
  <p><b>Sara Mannheimer  </b>16:20</p>
  <p>And did you write about this in your paper, too? Did you add, like, when you, did you talk through these ideas of doing double checks for language being spoken? And looking at individual users?</p>
  <p> <b>BSRO6  </b>16:35</p>
  <p>Yeah, and maybe this, I don't, I don't know, this can remain on a recording, but maybe like, there's a paper under review right now that I'm just trying to be sensitive to like, discussion points, can...</p>
  <p><b>Sara Mannheimer  </b>16:49</p>
  <p>I'l  make a note.</p>
  <p><b>BSRO6  </b>16:53</p>
  <p>But to to sort of like back this up, we actually ended up doing a complete study based just on, we took hurricane, there's multiple different ways you can gather tweets, and we just took a few different ways, so I can get tweets that are based on keywords, they say Hurricane Harvey, or maybe they were in the geographical region of Hurricane Harvey, and then just did a bunch of analyses that would be sort of typical of the social media analysis. Let's see how many people retweet and how many people post images and gifs and how many, like individuals overlap in these datasets. And they're entirely different.</p>
  <p>I mean, there's, they're, in almost every analysis we did, the way we gathered, the tweets was incredibly important. There're like 1% of users showed up in both. There's no overlap. I mean, every analysis we did was like, fundamentally different between the different ways that we gathered the tweets. So this was like, yeah. This was the sort of like, fol ow on that, you know, we realized in the course of this paper that like, yeah, this is this is a huge component of this, but probably deserves a little bit more attention than like, a discussion point. So let's try to like actually codified this in, like a more analytical way.</p>
  <p><b>Sara Mannheimer  </b>18:26</p>
  <p>That's great. Cool. Al  right. Um, let's move to data quality. Can you tell me a time during this research, when you considered the issue of data quality? So like, missing data? Or if there were busts in the data set, or the quality of other people's Twitter data collection, stuff like that?</p>
  <p><b>BSRO6  </b>18:47</p>
  <p>Yeah, yeah. I mean, this is like, this is like, I would describe this as the fundamental issue with like gathering Twitter data, at least from the perspective of like a data science from a very data science perspective, it's, it's very messy compared to other things. I'm just like a normal like good old fashioned data scientist. This is like as messy and as unclean and as unreliable as data sources come. So it definitely was a concern through all of it, specifical y, so things that were like known issues that we tried to deal with were that when you're rehydrating these tweets over time, you have, there's a, there's a loss, people either delete tweets, or they make their accounts private. And so there's this, there's a pretty like regular, just like decay of this data over time. And there is a particular, there's a paper that tested this specifical y. And they, they basical y just wait, they gathered a bunch of tweets, both like related to specific events, and then sort of just a broad sample of Twitter. And then five years later, they tried to go based on the idea, they tried to re, re-access the same data and they found I think it was about 75% of the tweets were stil  there. So they in five years, they lost 25% of their data. But by any means that they could find the remaining data was stil  a representative sub sample. So, so when you're looking at users, when you're looking at geographic distribution, anything that they were interested in testing, there was, it was a pretty nice sub sample. So it seemed as if that decay or if that that loss was pretty evenly distributed. And so that was pretty reassuring. We had losses, I think, in about the like, at somewhere between 85% on average of the tweets that I went to go access ended up being accessed. So that was one was just the some of it's just straight up missing. When you go to re-access it further on down the road. We did test for bots and remove bots using I think Indiana University has like a, an API for check you, you run a user through this, and it checks to see how many bots are in</p>
  <p>there. We, we attempted to remove them, and ended up not removing them in all circumstances.</p>
  <p>Because there was a lot of situational awareness type information that was present. Local weather stations wil  tweet the status of hurricanes or local news organizations wil  retweet a news headline, and that tends to be flagged as a bot. It's an automated Twitter account, but it is stil  providing situational awareness type information. So we did test for bots and reported the percentage of bots in them but didn't remove those. Because in certain circumstances that, our judgment call in this case was that that information is more useful and does provide context for what's going on, versus removing them where we sort of have this this gap that we have found.</p>
  <p><b>Sara Mannheimer  </b>21:59</p>
  <p>Again, did you like, what were your sources for understanding quality and figuring out the strategies for dealing with it? And then also can you... And yeah, I'l  stop there.</p>
  <p><b>BSRO6  </b>22:11</p>
  <p>Okay. So the primary sources for [strategies for addressing quality] were existing papers was was looking to the literature specifical y on gathering social media data around natural disasters and natural hazards. There's a, there's a reasonable enough amount of publications in that space that these issues, we were able to find these issues through literature reviews, and know that these were going to be coming. That was the primary source of trying to figure those out.</p>
  <p><b>Sara Mannheimer  </b>22:41</p>
  <p>Okay, and then you did you document this in your paper as well? Yeah.</p>
  <p><b>BSRO6  </b>22:47</p>
  <p>I think we have an entire appendix that's just like, here's what happened in the data collection process of things. Because there's just so many little steps like that. That are, our goal was to make it as repeatable as possible. I mean, all the code is published. But so you would understand if you would see if you read through the code, like, oh, you're getting rid of all of these things, or you're getting rid of all of these things. But our aim was to provide a more detailed justification of that, should someone be interested in replicating.</p>
  <p><b>Sara Mannheimer  </b>23:16</p>
  <p>Yeah. And so ultimately, you were like, you felt that these things you did to help to support data quality were enough? Like, yeah,</p>
  <p><b>BSRO6  </b>23:26</p>
  <p>Yeah. But very much a judgment call, like the very much a judgment call internally as to whether or not that was enough. Our.... Yeah, that was our eventual like result was to essentially look at by compute this, like analytical fingerprint for a given disaster, we we're interested in the sort of similarity of these fingerprints over time, or among different events. So is the fingerprint before different than the fingerprint after a hurricane? Or is this is a hurricane fingerprint, similar to an earthquake fingerprint?</p>
  <p>And so along the lines, we were sort of like checking to see the impact that it might have on the similarity of these fingerprints. And so if we removed the bots, does the fingerprint similarity remain the same? If we add them, so then if it doesn't, if removing them doesn't fundamentally change this, we're going to include them because then we think it does provide richer information. So a lot of it was tested sort of empirically against what we were aiming to see. Okay. So similar with some of the like, text removing, or text cleaning is another like obviously a huge part of this, do you leave emojis in, do I get rid of the word amp or like RT because this though the presence of those is, you know, indicative, it's comes in the text, but it may not necessarily be indicative of providing additional information other than</p>
  <p>that, it's a retweet. Do you filter out swear words? So a lot of this was was very much done to see okay, if I get rid of emojis do I have a less noise and more signals sort of in my result, and we could test things that way which was, which was good but the the when it was good enough was was totally a judgment call.</p>
  <p><b>Sara Mannheimer  </b>25:06</p>
  <p>And did you discuss that with your research team?</p>
  <p><b>BSRO6  </b>25:10</p>
  <p>Yeah, this was all all of us would sit there and look at these plots and say, I think it actually makes sense to do this one.</p>
  <p><b>Sara Mannheimer  </b>25:15</p>
  <p>Okay. Okay. Al  right, let's go to data comparability. So during this research, you did combine multiple datasets, right, that you had found elsewhere? How, like what issues arose with combining of data?</p>
  <p><b>BSRO6  </b>25:38</p>
  <p>That's a really good question. It's, one was the sort of scale or the scope. So it's unclear even now, where these differences in scope came, there are some natural differences in this scope. If you are one of the events we were looking at was an earthquake in Nepal. Twitter adoption is just, you know, orders of magnitude lower in in Nepal than it is in the United States, or in even in the United States, something happening. There's a river in Oklahoma, that floods, the Red River in Oklahoma floods pretty regularly.</p>
  <p>The Twitter adoption there in rural Oklahoma is a lot less than New York. And so Hurricane Sandy has this like, it's easy enough to hypothesize that the average person impacted by Hurricane Sandy is more likely to be on Twitter, than the average person in Nepal who's experiencing an earthquake. So one was this this concept of the place bed, the the context or the place specificity of the disasters we were actually studying. What we were finding is that, by and large people who were tweeting about an earthquake in Nepal, or people who were tweeting about the Ebola crisis going on in Africa, were not actively experiencing the Ebola crisis in Africa, they were commenting on it, a higher proportion of people who were tweeting about Hurricane Sandy, were actively experiencing Hurricane Sandy. So that was a very difficult attribute to deal with in this data processing. Because we're essentially we have datasets that, by their numbers, by their like structure are identical. That's, it's just tweets that have been gathered through a given period of time, but in their meaning, or in in what they sort of are representing the types of discussion or online discourse they're capturing are fundamentally different from one another. So we ended up getting rid of a lot of those that that were not as comparable so that the comparability was a lot about what we thought was being captured by the tweets as opposed to the sort of technical interoperability of the data sets.</p>
  <p><b>Sara Mannheimer  </b>27:42</p>
  <p>Yeah. How did you identify which, which tweets were commenting on the disaster rather than experiencing it?</p>
  <p><b>BSRO6  </b>27:53</p>
  <p>A lot of it was just looking at like Twitter's annual reports, they have a pretty good description of their monthly active users by country by time period. That's, that's pretty easy to find information. So we would go through and just look and say, how many monthly active what's the percentage of the population of people in Nepal in 2009, that have Twitter and it's, so let's, that's probably not people experiencing a disaster, we'l  strike that. So we used like, a proxy was, was how much Twitter was used in a given area in a given country at roughly the period of time when the disaster was happening.</p>
  <p> <b>Sara Mannheimer  </b>28:34</p>
  <p>Cool. Okay. One thing I meant to ask you before was did with quality, I guess, in comparability too, but like, did you encounter when you were trying to reuse these lists of tweet IDs that you had found? Did you encounter any, like problems with the way that other people had documented their data or missing information about the data that made it more difficult to use?</p>
  <p><b>BSRO6  </b>28:56</p>
  <p>Yeah. There, this was something that is, it's it's not easy. But it is possible to do after the fact to go through this list of tweets. And I, it's not hard to see that everyone contains a keyword. And so this was probably gathered because they just look for everything matching a key word, or some of these are just total non sequiturs that aren't related to this at al . But if you go to the profile of the person, you can see that they are located in Houston during Hurricane Harvey. So we were able to sort of intuit that over time that they may be gathered in different ways. But yes, there were no clear reporting guidelines, there's by no means a standard as to it. It took us having to go through and sort of manually double check all of these to make sure that we could find a reasonable have a reasonable guess as to how this data was gathered or what the sort of the sampling rates are, how much data they got, we had to go through and figure that out after the fact there was no documentation for it and that was a real sore spot, so to speak.</p>
  <p><b>Sara Mannheimer  </b>30:01</p>
  <p>Yeah, were any of those, that data you were using, did you find it in like specific data repositories like....</p>
  <p><b>BSRO6  </b>30:09</p>
  <p>Most of them were, like, attached to githubs that were like, attached to papers. And I think in three or four cases, we were able, there's a program called Social Feed Manager, which is like a software for scraping tweets, that does a pretty good job. And that has what I would, I don't know if archival quality is like the right term, I'm not sure if that term has specific meaning, but it has very high level very detailed reports as to what filters were being used all of the query attributes that were being applied to it to pull tweets out. So those were the like, the best of the best, because I know exactly what was being used to do that.</p>
  <p><b>Sara Mannheimer  </b>30:50</p>
  <p>That was developed by librarians.</p>
  <p><b>BSRO6  </b>30:54</p>
  <p>Yeah, it definitely. It was it was excel ent for like our purposes to be like, 'Okay, I know exactly what's going on here.' Where especially older tweet data sets, it was someone very, a lot of them felt like it was someone who had written their own custom code to like, just go, just go try to get as many of these tweets as we possibly can. Yeah. So yeah, that was really specifical y those were like the best examples, but by no means was that a universal experience.</p>
  <p><b>Sara Mannheimer  </b>31:21</p>
  <p>Okay, great. Very interesting. Al  right, let's move to informed consent. So can you tel  me about a time during your research when you considered informed consent?</p>
  <p><b>BSRO6  </b>31:32</p>
  <p>Yeah, I mean, it's like a little creepy. Like, when you first like open one of these, you take the tweet IDs, you hydrate them, and you're like, 'Oh, this is just a person and you live here, and you just tweeted</p>
  <p>about your house.' And that's just on the internet now. Ultimately, the, like any concerns we had does, from like, a moral sort of invasive privacy standpoint, it feels a little icky. But in terms of like, you know, what actual regulations are there, we were again sort of leaning on the Twitter private policies, the policies, the terms of services that are Twitter's Terms of Service and how they govern the use of these developer accounts that you have to have to access this data. That was what we kept going back to to say, 'Okay. According to these rules, it is okay for me to publish this data.' In no cases did we use like actual tweets, content, just sort of, like, out of respectfulness didn't feel like the right thing to do to just publish the content of a tweet, despite it being public and being aboveboard to do. But that was it was the terms of service that sort of like governed that.</p>
  <p><b>Sara Mannheimer  </b>32:40</p>
  <p>Tell me more about that, like feeling that it wasn't right to publish. Where did that come from? And then how did you decide among yourself that that was going to be like a part of an internal policy that you made?</p>
  <p><b>BSRO6  </b>32:53</p>
  <p>That's a good question. I don't know like where specifical y it came from, other than to say, like, you just doing some sanity checks, just sort of like, 'Alright, I, I, you know, little grad student. And I've, I've run my code overnight, and I wake up the next morning, and I go into the office, and I like, Oh, look at all these tweets that I have. And I can just kind of scroll through them. I'm like, Look, let's see what I just pulled off the internet.' It is very invasive. I mean, people would be talking, especial y in the data sets that were gathered in a more sort of dragnet fashion where it was maybe any tweet in a geographic area, or any tweet in a given time period that was in a geographic area versus tweets that were containing a keyword for a disaster. Those tended to be pretty intuitive, and pretty standard, you can you can, those were a lot more predictable. But if it was just everyone in this region during a disaster, you got a lot of things that weren't disaster related, despite them being public. And so with the feeling arose from seeing a lot of the things that from just experiencing, or seeing a lot of the things that people would post, and it is not relevant to what we're doing, per se, this is not like necessarily informing our analysis, but it is data that shows up as a part of it. But no, nothing that we, everything is essentially de, deidentified or anonymized in our algorithm. So we pull text, but it's no, it's never attached to a specific user once the processing is done. So when we're actually processing or we're actually computing things, all of that is done on, on a large corpus of text, not on a specific user's individual tweet. So when we were publishing it, there is no requirement in the slightest that we have identifiable information that we didn't need to publish specific tweets, none of that was required to improve our analysis. So it's, if we do not have to this is absolutely not something we should do.</p>
  <p><b>Sara Mannheimer  </b>34:49</p>
  <p>Okay, and so you didn't publish any, like, sample quotes or anything from...</p>
  <p><b>BSRO6  </b>34:54</p>
  <p>Yeah, we had, you know, one which sort of described we used a couple that were essentially made up, we would just I would tweet something. And then we would run it through to show like how this the algorithm pulls out this word to mean this and this word to mean this, any such circumstances of those, I would just tweet and just do it myself. So we had one sample, but it was, you know, a purely orchestrated made up example, that was just a tweet that had been put out on the internet.</p>
  <p><b>Sara Mannheimer  </b>35:25</p>
  <p>Okay. And was there anybody that you looked to for guidance, to come to this strategy?</p>
  <p> </p>
  <p><b>BSRO6  </b>35:34</p>
  <p>Not particularly. I This was I think there was a team of four of us that were working on this. And so this was just an internal discussion as to sort of like what felt like the right thing to do. Our, our group and the people who were doing this had a bit of background on working with like, power company data. So again, it's like resilience of communities to disasters, we had previously together done a study, looking at power outages in response to like Hurricane Katrina. And this is like really sensitive, fully identifiable data, you have a name, you have an address you have when their power went out when the power came back on. And in circumstances like that, like the sort of standard protocols were to de-identify everything immediately and only work with anonymized or deidentified data. And so that was a bit of experience that we had had that, you know, if possible, if there's, if there is something that could in any way be considered sensitive, it makes sense to de, de-identify and go on and use that. So that I think also contributed is some sort of historical work, having done work with PII led to this idea that maybe this isn't like PII by the letter of the law, but it is PII sensitive adjacent. Yeah. And so it felt like a right thing to do not necessarily if it was governed by something.</p>
  <p><b>Sara Mannheimer  </b>36:56</p>
  <p>Yeah. Okay. Sensitive adjacent, I like that. Okay, and the next issue is privacy and confidentiality. So this is closely related. But was there a time during your research when you considered the issues of privacy? How and then how did you deal with those? How did you conduct the de-identification? And how did you use the aggregation in order to support your conclusions, but while supporting privacy?</p>
  <p><b>BSRO6  </b>37:29</p>
  <p>Yeah. So the analysis that all of the algorithms that we developed are just based on text. So our sort of privacy steps were to take this series of tweet IDs, rehydrate them. So we've got all of the information we need, just extract the, the raw text, and then that would be the only thing that would leave, we had a particular secure, we call it the secure server, it was the one that was just a workstation that was doing all of the lifting. And then anything that we would take off of that was only text. So our de, and that was our de-identification processes, then for a given disaster, I have a series of tweet texts, but it's not linked at al  to an account, it's not linked at all to a person, it doesn't even have a tweet ID attached to it anymore. So that was what we, again, you know, we could have just used from, from our understanding of the terms of service, we could have used all of it, and it would have been okay. But that felt like the sort of, if we didn't need it, it wasn't important to, to bring that information along. Let's try to use as little as we can. And there was one specific instance. This was a disaster that I think we ended up not including in the final paper. But there were situations in which there would only be...It was a very small earthquake in California in like 2009, it's really in the fledgling, Twitter was super young at that point.</p>
  <p>And the data that we had was really small compared to some of the others, right, Hurricane Sandy has like 13 mil ion tweets, I think totally it's 250 mil ion tweets, and this had like 25,000 tweets. And so when we're looking to try to match, like maybe this tweet matches institutions and political, there were a couple situations in which there was like one tweet that matched those. And we didn't, we didn't end up including this. This was not published. We didn't we didn't write this analysis. But there was a consideration like if there is there is one tweet that matches these things. I guess in theory, you could back out which one it was. And it stil  felt a little identifiable in the way that like, you know, the census doesn't publish data, if it shows that there's a there's a lower limit on income. So they like if there's only one house making a given income in a given neighborhood, they won't report that so you can't go 'Oh, it's these people that make this income.' So you know, sort of in a similar vein, if it were getting to the point where there was only a few tweets that were like ending up motivating our analysis, we got rid of it. But that was another time where that was sort of like, potential questionable privacy issue.</p>
  <p><b>Sara Mannheimer  </b>40:17</p>
  <p>And did you... Okay. And then with your data publication, you didn't, did you publish the text of the tweets, or you also just published tweet IDs?</p>
  <p><b>BSRO6  </b>40:27</p>
  <p>We just attach tweet IDs in our sort of repository. I think that's I think that's a Twitter Terms of Service thing is I'm only allowed to share tweet IDs. And then in our paper itself, there was no text. So it's just....</p>
  <p><b>Sara Mannheimer  </b>40:41</p>
  <p>The analysis is with the text, but then you publish the tweet IDs, and then you have your code that shows how you went through it to pul  it.</p>
  <p><b>BSRO6  </b>40:48</p>
  <p>Exactly. I'm not actually I'm not sure if we published the hy-, like the code that takes the tweet IDs and hydrates the tweet IDs. But we stil  think it starts with let's assume you have a large bit of text, and then like our code that we've shared starts there.</p>
  <p><b>Sara Mannheimer  </b>41:01</p>
  <p>Nice. Okay. Did you...So what did you think about what the Twitter users' expectations of privacy were, you know, thinking about the terms of service and like how the users understand those?</p>
  <p><b>BSRO6  </b>41:20</p>
  <p>No, truth be told, and probably should have, I think this is something that I like to think I'd be more like aware of, or mindful of now. I just think that we did this work in 2017, into 2018. And so it is, like, in my personal life, privacy has become like a larger concern over time. So my hope would be now that that would be the case. But I think at the time, that was not the expectation of what a user was hoping their data would be used for, really didn't didn't play into it.</p>
  <p><b>Sara Mannheimer  </b>41:53</p>
  <p>Al  right. I know, it's funny how I feel like our perceptions about online presence and social media at change, like every three months. There's so much... Yeah.</p>
  <p><b>BSRO6  </b>42:06</p>
  <p>I remember being like a teenager, and like sneaking out of the house and going to a party. And just like the next day, like don't post that on Facebook, right? Like, you know, 'Don't tag me in that picture on Facebook,' and like, and thinking about then versus now, where if someone did that, that would just be that would be socially like a really bad, you would never do that you would never do that. Because they would have social repercussions. And as a society, we've learned like, now you just don't tag embarrassing pictures of your friends when that we had to learn that at a certain point. That was like not something that we had figured out. So yeah, I think I hope that my like user understanding of privacy has grown. But hopefully everybody's has.</p>
  <p><b>Sara Mannheimer  </b>42:45</p>
  <p>Well. And my, I hope that eventually we have like policies that can govern this stuff, because it's a little ridiculous that you were relying on individual researchers like you to be like, you know, it didn't feel right to do that. So, like the IRB should govern that or that, you know, so. I'm letting my own thoughts come through. Al  right. Our last question is about intellectual...Our last official question is about intellectual property. Was there a time during the process of this research when you thought about intellectual property concerns? Yeah.</p>
  <p> </p>
  <p><b>BSRO6  </b>43:26</p>
  <p>I don't think so.</p>
  <p><b>Sara Mannheimer  </b>43:31</p>
  <p>For example, when you were reading the terms of service, Twitter's Terms of Service, like, it sounds like you just really went through those and followed those to the letter of their law.</p>
  <p><b>BSRO6  </b>43:42</p>
  <p>Yeah, yeah. What, what I know, is a concern, or again, this is sort of like the, there are people so this was what would have been what would have been a concern with regards to intellectual property is like the, the nuances or the specifics of the algorithms that we developed. So again, for example, this group had previously done work trying to predict the number of power outages in a given place. There's now a lot of companies that sel  that as that's their, the algorithms, they wil  predict power outages, and they wil  sell that information to cities, and the, the underlying tools they use to predict that are their IP. And so our concern, I think, was like if this, let's say I could make a really great prediction with social media data about how the community would respond to disaster XYZ, that the process of making that prediction was our concern for IP. And so you know, maybe someone was going to steal our algorithm and do something with it or use it to they could use that as the basis of a company that they could try to sell something. But that I think is mostly covered by University IP policies. If you, I'm a student at this university working with a professor at this university, I think the university, my understanding was that my university just owned anything that I created while I was working on there. And it would be the university who would work on those sorts of like, IP disputes on my behalf. With regards to like the actual content from Twitter, now, there was nothing specific that was like, privacy was a much larger concern than intellectual property.</p>
  <p><b>Sara Mannheimer  </b>45:34</p>
  <p>Yeah. Okay. Good. Great. So the last question is, is there, were there any other issues or challenges that came up during the research that I haven't asked you about?</p>
  <p><b>BSRO6  </b>45:48</p>
  <p>That's a really good question. I think the largest challenge is, is more of like a, like an academic and sort of process one, which we've already touched on a little bit, but it is like the, the interface between having the sort of technical and like computational know how to access all of this data, process al  of this data in a way, which has some level of integrity, then ultimately, you're, you're ending up with a data set that is very much the result of human processes. And so it was like, I had the expertise to go on the internet and pull al  of this data up. But I have no expertise on how on theoretical or sort of conceptual understandings of how people process information. And it was tough to find people who were to, it was a tough collaborative effort to try to find people who could be at this intersection in to be programmatic enough to pull 150 mil ion tweets from Twitter, the Venn diagram of the people who can do that, and the people who have like, firm social scientist training and understand what this data means is vanishingly small. And so it was a lot of collaboration and a lot of discussion to try to create a team that could sort of balance both of those, I think that was a very big step, because it's very easy to say, 'Well, I pulled all this data out, I'm sure I can find something to do with it.' But I have no idea if that represents what any sort of way that a human might actually think or act. And so I need to involve that expertise as well. So that the technical hurdle was limiting. So you had to have the technical skil s. But that did not necessarily mean you had the formal training in a more social scientific side, to understand what that data meant? That was a huge challenge.</p>
  <p><b>Sara Mannheimer  </b>47:47</p>
  <p>How did you find the communication faculty who you reached out to?</p>
  <p><b>BSRO6  </b>47:52</p>
  <p>Oh, pure luck. I had a, I had a committee member who was like 'Oh, you should talk to this person.' And we started speaking like, wow, this is like really great that we ended up here. It was an issue that we knew we were going to come across. And so eventually, we're going to start just soliciting help. But that was yeah, it was luck that we got connected with this person in particular, and certainly very fortuitous that that happened.</p>

  <h2><b>Sara Mannheimer </b></h2>
  <p>And did they end up being a co-author or an advisor?</p>

  <h2><b>BSRO6 </b></h2>
  <p>They were yeah, like sort of more of an informal mentor. I think they're mentioned as an acknowledgement. But they were because the nature of the relationship was very much 'Hey, I have these things. Does this make sense? Can you like, you know, sanity checks some of these?' Is it okay, well, here's maybe some papers you should read. Here's maybe some like, here's a textbook that's going to help you like catch up to speed on this. More so than someone being like directly involved in sort of contributing in an authorship way.</p>
  <p><b>Sara Mannheimer  </b>48:53</p>
  <p>Awesome. Cool. Well, thank you so much. This was so fascinating. It's gonna be a great addition to my, my research, so I really appreciate it.</p>
  <p><b>BSRO6  </b>49:48</p>
  <p>Yeah, absolutely. So it was great to meet you.</p>
  <p><b>Sara Mannheimer  </b>49:50</p>
  <p>You too. Have a good day.</p>
  <p><b>BSRO6  </b>49:52</p>
  <p>Yeah, you do the same.</p>
  <p> </p>


name: BSR_07
filename: Mannheimer_BSR07_Tra.pdf
created: 2024-07-11 14:35:02.376759
text_direction: LEFT_TO_RIGHT
contents:
  <p> BSR07_transcript_deidentified</p>
  <h2><b>SUMMARY KEYWORDS </b></h2>
  <p>pins, data, pinterest, misinformation, question, feel, people, research, thinking, coding, context, published, intellectual property, url, guess, qualitative, posted, issues, informed consent, study <b>SPEAKERS </b></p>
  <p>Sara Mannheimer, BSR07</p>
  <p><b>Sara Mannheimer  </b>00:00</p>
  <p>Okay. Um, and so I have I sent you the interview guide, you can look at that if you like, or we can just kind of talk through it. But I told you a little bit about my research. The main research question is how can data curators best handle qualitative and big social data to support ethical, epistemological and legal data sharing practices. And so I did a literature review and revealed six key issues that pose challenges both for qualitative reuse and for big social data. And so the interview is structured around those six issues, which are context, data quality data comparability, informed consent, privacy, and intellectual property. It wil  take about an hour. So yeah, there wil  be I have some intro questions about six common interview questions al  together, one on each of those topics, and then we'l  do a little wrap up. Okay, any questions before we begin?</p>
  <p><b>BSR07  </b>01:08</p>
  <p>Um, no, I don't think so. That sounds good.</p>
  <p><b>Sara Mannheimer  </b>01:12</p>
  <p>Okay. So tell me about you and the type of research that you do and the type of data that you produce.</p>
  <p><b>BSR07  </b>01:20</p>
  <p>Okay. Um, so yeah, you you identified, you know, a particular study where I used big social data [from Pinterest], but I actually use a variety of different sources of data and methodological approach approaches. So my research interests I'm, I'm a PhD student, and my research interests have to do with misinformation, news, literacy, and trust in use. So actual y, a pretty a pretty broad area. And, and I think it's important to use, you know, whatever is the the tool that is most appropriate to the research question. So I'm a mixed methods researcher. So I use content analysis, surveys, experiments, focus groups, interviews. So qualitative and quantitative.</p>
  <p><b>Sara Mannheimer  </b>02:18</p>
  <p>Yeah. Oh, interesting. I feel that wil  be a good viewpoint to discuss these questions.</p>
  <p><b>BSR07  </b>02:24</p>
  <p>Oh, cool. Okay. Al  right. Excellent. Yeah. You know, actually, like one thing that I did want to bring up, I guess, that I found like a little. I think it might just be like a difference in, you know, like us coming from different fields that we kind of think of maybe some of these concepts and classify things a little differently. Like, when I first read what your research was about, I was, like, a little confused by the term. Like, what? I'm trying to remember what you call it now. You call it qualitative...</p>
  <p> </p>
  <p><b>Sara Mannheimer  </b>02:56</p>
  <p>Data reuse.</p>
  <p><b>BSR07  </b>02:58</p>
  <p>Okay. Yeah, I guess, because to me, I think that like, at least like in journalism and communication, we talk about qualitative and quantitative methods. But a lot of times, the source material I think of is kind of agnostic, like, you know, that an interview transcript, for example, you could look at in a qualitative or quantitative way. So just I think, just to give you an idea of like, where I'm coming from, like, how I approach these issues, I sort of think of like, you know, you have your, you have your data set. And then, I mean, not all data could go either way, but some could and, and actually, in the case of this Pinterest study, we did do some, some, what could be considered, I mean, it was kind of a weird hybrid.</p>
  <p>We did some analysis, that was definitely quantitative. And then we did some that was like, we kind of deductively or inductively, rather, came up with categories for like, types of misinformation, and then and then quantified it. So it was kind of it was sort of like a mishmash of, of qualitative and quantitative approaches.</p>
  <p><b>Sara Mannheimer  </b>04:05</p>
  <p>Yeah. From the librarian perspective, we think a lot about like, the material itself, you know, yeah, I guess. I think where we're disconnecting here is I'm thinking of the data as qualitative data, and yeah, whatever. And the big social research as using social media data. And so for my research, I'm not so much interested in how what methods you use to analyze the data, but sort of like how you find the data and how you think about the people behind the data, thinking about them. So that's why the data we use, it's like, if you when I publish these transcripts in a repository, how do the how to data curators and librarians like give, ask for the right documentation and work with the researcher to create the right information package that can allow the data to be reused and understood in the future. So I'm really coming from it from like a "stuff"—a "data" perspective. Like the data alone <b>BSR07  </b>05:14</p>
  <p>Yeah. Yeah.</p>
  <p><b>Sara Mannheimer  </b>05:15</p>
  <p>It's not as much of interest to me like, what research methods you use on the data.</p>
  <p><b>BSR07  </b>05:22</p>
  <p>Okay. Okay.</p>
  <p><b>Sara Mannheimer  </b>05:23</p>
  <p>So I guess, as we talk through the questions, maybe we can continue to discuss this, because that's interesting to me as I write up my results. That was really helpful to hear.</p>
  <p><b>BSR07  </b>05:35</p>
  <p>Okay, cool. Sure.</p>
  <p><b>Sara Mannheimer  </b>05:37</p>
  <p>So can you tell me about a recent time and I sent you this in an email where you either collected big social data for research or reused big social data that was shared or prepared your data, your big social data for sharing?</p>
  <p><b>BSR07  </b>05:52</p>
  <p>Yeah. Yeah. So I think like, I think the Pinterest study is probably the best example of that. And I hope it's okay. It's not super recent, like I collected the data at the end of 2018.</p>
  <p><b>Sara Mannheimer  </b>06:05</p>
  <p>That's okay. As long as you remember.</p>
  <p><b>BSR07  </b>06:11</p>
  <p>Okay. I hope so.</p>
  <p><b>Sara Mannheimer  </b>06:13</p>
  <p>I went all the way back to 2018. When I was doing my search, so.</p>
  <p><b>BSR07  </b>06:17</p>
  <p>Yeah. And it well, it probably popped up also, because it's like, it was only published what last year? So you know, of course, these things take a while. But yeah, that's so that's probably the best example because I haven't, I've done a little schoolwork that involves some big social data. But that's the that's the last thing I did that was like, that I published or presented that used it. So basically, we we were interested in the incidence of misinformation about [a topic] on Pinterest, and, and also just some of the qualities of of the misinformation and you know, what sort of... what kind of topics within [the broad topic]? How did it relate to whether it was, did it relate to [related topics]. So just trying to, like get a sense of like, what, what does misinformation about [the topic] look like on Pinterest? So we used [a commercial tool], which is um, I'm trying to remember if it's free or not, I think I may have paid a nominal fee for it. It's a scraping software. So it, it allows you to put in a URL and create a script that it just like goes in and iterates. So I was able to put in like search for [the topic], and it would go in and like repeatedly search. Or sorry, it would it would search one time, but then it would like repeatedly scrape the URLs of all the resulting pins. And then I did that on three different days. So um, yeah, I don't know, what else what else would you like to know about it?</p>
  <p><b>Sara Mannheimer  </b>08:21</p>
  <p>I think that's good. Um, so that's the data collection method. Um, was this part of grant funded research where you had a data management plan or a specific way that you were required to handle the data?</p>
  <p><b>BSR07  </b>08:34</p>
  <p>Um, we did. So my co author was from [a different university], and we did have a small grant from the university, but I think it was I got $500. So I think it was $1000 total. And but I there wasn't any like stipulation on that. They didn't say anything about data collection or management.</p>
  <p><b>Sara Mannheimer  </b>08:54</p>
  <p>Okay. And then did you publish any of the data from your example? Um, anything interesting...</p>
  <p><b>BSR07  </b>09:02</p>
  <p>We did just include one kind of, I don't want to say example pin, because it was really kind of like the most egregious misinformation. So one piece of misinformation in a pin that was like an il ustration in the an article that we published, but we did not like publish the data set.</p>
  <p><b>Sara Mannheimer  </b>09:29</p>
  <p>Okay, let's move to our first question about context. So I have a little blurb to sort of help understand what we mean by context and this question. Halavais in 2015 suggested that "When we collect data from social media platforms, just like when we collect data in traditional spaces, context matters.</p>
  <p>However, the context of a social media post may be absent or difficult to understand. social media posts are by nature short pieces of text, images, videos, etc, that are taken from a larger context of personal and public life. And then the out of context effect can be compounded when data are masked at a large scale." So kind of thinking about this issue just with social media data in general. Can you tell me about a time, if any, during your research with Pinterest, when you considered the issue of maintaining and understanding the data's context? So like information about the community where the data was collected, or contextual information about the people who had pinned each of these pins?</p>
  <p><b>BSR07  </b>10:36</p>
  <p>Um, yeah, actually, if you don't mind, let me just I'm going to refer back to my paper to I do need a little I do need a little reminding actually, sometimes. We we kind of like we and, and just as background, like we did kind of toy for a little bit when we were coding with, like, different ways of representing the source of the information like, were we going to, were we going to, or I think we we coded... Actually, for the, let's see, the source of the... actually, let me open it up, because I don't want to get this wrong.</p>
  <p>And I think this only, yeah, I don't want to like linger on this too much because I'm, I think this kind of answers your question, but I'm not sure. So. But let me just, let's see, open up here coding. It's my coding spreadsheet. Okay. Yeah. So there was, um, there was like the main source. Actually, you know, what, it'd be better if I opened up my code book, because that would put it in context.</p>
  <p><b>Sara Mannheimer  </b>12:05</p>
  <p>And these are supplemental materials? And you're just opening that?</p>
  <p><b>BSR07  </b>12:10</p>
  <p>Um, yeah, actually, that's a good point. Because we did. Yeah, we did supply the codebook as a supplemental material. Let me see. Sorry, I just like, I saw your questions ahead of time. And then I just thought of this, and I was like, oh, maybe I should share this. Um, okay. Hang on a second. Okay, here we go. Okay, so we coded for the type of website that the pin linked to. And we tried to capture was it like a mainstream media website, social media site, blog, [lists several other types of websites], or other site. And then I think we had another, yes, then we had the source of the factual claim in the post. So</p>
  <p>[examples of where a factual claim would come from]. So we were trying to kind of get a sense of, you know, how that how the source of the information and the source of, and the actual website link to in the pin, how that kind of informs what people are saying in these pins. So like, that's a sense of context that we were trying to get. And in the end, I think it ended up being actually kind of, like, it's weird. I think that it was kind of, we only used, we didn't use both of those, we only used one of those in the, in the final paper, because I think it was just a little bit confusing to try and like, communicate all that information in a very short research paper. But I guess like, but I think the context has a lot of other meanings in this. So like, and I think there's a lot of ways that we didn't account for context. So we, you know, on the one hand, like, pins are kind of atomized and people post them with the expectation that they're going to be pinned to different boards. And I think that people are kind of thinking of them as individual. And so I'm less concerned about the context of the person creating it, but I'm more concerned about the context in which people see it. And that was something that we just kind of had to acknowledge as a limitation that, you know, we created fresh Pinterest accounts, because I figured like, you know, it's better, you can't, how are you going to account for the fact that when some people see it, and they've let's say, done a lot of searches about [lists several topics related to the research topic], this is going to influence the kind of results that you see, or if they've pinned those some of those things that's going to influence what they see.</p>
  <p><b>Sara Mannheimer  </b>15:37</p>
  <p>So because of the Pinterest algorithm?</p>
  <p> <b>BSR07  </b>15:40</p>
  <p>Yeah, exactly. Right. So like I was concerned. So I couldn't really see a way around this. Like, I think the best solution was just to create a fresh account and say, like, Look, this is what you see, with a fresh account. If you've been searching for these things, you're going to see probably more things like this. But it's definitely a concern like and it was brought up by reviewers, it was brought up by [a colleague] that like people were really kind of concerned that like, it's not the whole picture of how people are, are seeing this, or the amount or type of misinformation that they're seeing. So that was like a contextual question that came up for us. Yeah.</p>
  <p><b>Sara Mannheimer  </b>16:27</p>
  <p>Did you consult with anyone or consider other research projects or other literature when you were considering these issues of context? To try and come up with your solutions?</p>
  <p><b>BSR07  </b>16:40</p>
  <p>You know, like, I don't I don't, I don't recall exactly consulting with anyone. I think my my approach was kind of it was actually kind of influenced by a real y ad hoc research project that [another researcher]</p>
  <p>had done. It was like not published research was just like something [they] put on [their] blog, where</p>
  <p>[they] created a fresh account, and [they] saw how quickly this information populated and I guess that's kind of what made me think like, well, that's, that's a useful approach, like that kind of gives us a baseline. But, um, yeah, and then I would say, also that I was influenced by the, the peer reviewers, they kind of gave us some references that helped to helped us to explain, you know, how, how what we saw might be different if people had that context, and put some nuance on that.</p>
  <p><b>Sara Mannheimer  </b>17:42</p>
  <p>Okay. And let me look at the journal. So was it mostly like [scholars who study the topic of the Pinterest search] who were reviewing or was it journalism and communications people? Do you know?</p>
  <p><b>BSR07  </b>17:58</p>
  <p>It was actually it was blind. So yeah, I'm not sure. I think I'm kind of thinking from some of the references we got that they were, at least one of them was like communication. But maybe, maybe it was a mix. But yeah, I can't know.</p>
  <p><b>Sara Mannheimer  </b>18:11</p>
  <p>Yeah. Okay, cool. Let's move on to data quality. Can... and I guess, I feel like I should have said data quality and data, like trustworthiness, because that's kind of where I'm what I'm trying to get at here. So can you tell me about a time, if any, during this research when you considered the issue of data quality, or trustworthiness. So, for example, you know, what you were seeing on your timeline, or your Pinterest account, any missing data, or bots, or bias?</p>
  <p><b>BSR07  </b>18:48</p>
  <p>Um, there was a little bit of an issue with missing data. So like [the scraping tool], would, would spit out all these URLs. And, you know, I think probably at the time that it did the search, those were all URLs that lead to actual pins, but then by the time, I mean, I didn't leave it that long, but like a few weeks, maybe later, I was clicking on them to download all of the posts and to code all the posts, like some of them no longer they were there broken links. And that's probably just a, I mean, I suspect that could be a function of Pinterest moderation, like maybe it actually, maybe it found some egregious misinformation and said, nope, you're going down. So maybe it wasn't a huge like, it wasn't a huge amount of data loss. There were stil  plenty of stuff to code. So that wasn't it. And I don't feel like it was</p>
  <p>just it really distorted the sample because again, it wasn't like it wasn't a huge amount. And then, I mean, I think the issue of bots is interesting, we didn't really consider or try to account for how much of the pins might have been posted by bots, I kind of think that it's, it doesn't really matter for our research questions that much like if, if that misinformation is up there and can be easily found in a search, and people can see it and be influenced by it, I'm not sure how much it matters that if it was posted by a bot rather than a person, so it wasn't something that we really thought about.</p>
  <p><b>Sara Mannheimer  </b>20:25</p>
  <p>Yeah. That makes sense. And when you were thinking about the broken links, and which of these quality issues were really important, and which maybe you could... were okay. Did you consult with anyone there? Were there any resources that you used or other studies that guided you?</p>
  <p><b>BSR07  </b>20:53</p>
  <p>Um, no, not really. I mean, I think that my co author and I just talked about, you know, how much of the data is this? Is it? Is it going to bias our data at all, but, again, if anything, I feel like, you know, if, if the most likely explanation for those things being taken down is that they were taken down by moderators, then that means that we are probably under estimating the amount of misinformation so I don't yeah, I think that we concluded that was not an issue.</p>
  <p><b>Sara Mannheimer  </b>21:27</p>
  <p>Yeah. Okay. Great. Alright, data comparability. So. Um, during your example, did you compare or combine multiple datasets?</p>
  <p><b>BSR07  </b>21:43</p>
  <p>No, we well, I mean, we just, we did collect on different days. So I'm not sure exactly what exactly you mean, by can combine it like it. So we collected on I believe it was three different days. And, and put those together. So I don't know. Does that count? Does that sort of?</p>
  <p><b>Sara Mannheimer  </b>22:01</p>
  <p>I guess that's not quite what I'm getting at. Okay. Okay. Like, if you combined, tried to combine data from different platforms?</p>
  <p><b>BSR07  </b>22:09</p>
  <p>Oh, no.</p>
  <p><b>Sara Mannheimer  </b>22:10</p>
  <p>Yeah. Okay. And let's see, and or did you consider why.... I don't think this is really applying for your situation. But did you consider like comparability or interoperability of your data set? Like, did you think, did you save the Pinterest data that you scraped? And did you think, oh, maybe I wil  scrape more in the future, or I'l  pull some information on Facebook to do a future study, follow up study, anything like that?</p>
  <p><b>BSR07  </b>22:40</p>
  <p>I guess that's kind of, I guess, that was sort of in the back of my mind that I might do another study on another platform. But yeah, so that's, that's a good point, I guess, because it was in the back of my mind, and really not at the forefront. I, it's probably not something that I, I didn't really take steps really overtly to think about how, how this would, yeah, whether it would be comparable.</p>
  <p><b>Sara Mannheimer  </b>23:11</p>
  <p>Tell me more about like the type of metadata, or you know, what comes back from the scraper? Or what do you end up with? Like, does it give you the image? And the links out? In what form does it come back? Is it like, a CSV file? Or?</p>
  <p><b>BSR07  </b>23:27</p>
  <p>Sure, yeah, so actually, let me I'm gonna open it up. And I can also send this to you if, if it's of interest, I'm sure I'm just I'm going to open it up. Just to remind myself what, what I got. So let's see scrapes.</p>
  <p>Here we go. Raw scrapes. Um, okay. Yeah. So it gave me it's yeah, not a lot of not a lot of columns in this data set. So basically, it's a CSV file. And there's [a number of] columns, there's the URL for the pin itself. And then there's the URL for the image in the pin, which is also like, it's also a URL that starts with pinterest.com. It's just, you know, if you want to look at that, specifically, you know, it would have been great if it had given me a lot more of the things that I was coding for. But that's basically what it could do. So then I had to go in, click on each URL to do a screen grab. And then save it. So it was a lot of manual labor.</p>
  <p><b>Sara Mannheimer  </b>24:38</p>
  <p>I see. So you use the scraper to identify the pins that were of interest to you. And yeah, and you did a manual check after that.</p>
  <p><b>BSR07  </b>24:46</p>
  <p>Yeah, exactly.</p>
  <p><b>Sara Mannheimer  </b>24:48</p>
  <p>Okay. And did you end up, did you keep adding to the spreadsheet with like, the date that it was pinned or the user name or any of that or was it just it ended up just being the pin?</p>
  <p><b>BSR07  </b>25:01</p>
  <p>Um, yeah. So I mean, the, the pins I collected as JPEGs I, and then I used the CSV that the scraper gave me to basically, basically be the start of my codebook. And then I, yeah, I had, let's see how many codes did I have? I had a whole bunch of codes. And we didn't end up using everything in the end, but it was like [a number of] different things that we coded.</p>
  <p><b>Sara Mannheimer  </b>25:46</p>
  <p>Nice. And it's... you shared the guide for coding, but did you didn't share like your codes, like the raw codes that you had created? Is that right?</p>
  <p><b>BSR07  </b>26:03</p>
  <p>Yeah.</p>
  <p><b>Sara Mannheimer  </b>26:03</p>
  <p>Okay, just checking that I wasn't missing. Okay, cool. Al  right, let's move on to so I've grouped these, like the first three are, I call them epistemological issues, sort of just more about the research itself. And then the second three are about ethics and legal issues. So we have informed consent, privacy and intellectual property, the three that are remaining. So let's talk about informed consent. Can you tell me about a time, if any, during your research, when you considered the idea of informed consent for the participants, so to speak?</p>
  <p><b>BSR07  </b>26:39</p>
  <p>Um, I mean, I didn't, I didn't really consider it. I think that we, you know, the, how we've been trained in this program, is that when you're doing content analysis, I mean, right or wrong. Like I think, I think you can, you can ask questions about this, for sure. But I think we're sort of trained that it's like, you're not studying people, you're studying content. And, you know, therefore, there's no need to obtain informed consent. Like, I think that's a little flawed, because obviously, people created the content. And people have certain expectations of like, how they think things wil  be used or not used. I think that like, if I had, I mean, I think you know, what, there is one point where I probably where I did think about it a little bit, and that is in the example pin that we published in the article. And that included, it did include the website, like, so it's got it, you know, it's got a subject line, and then the, the website that, that the, that the pinner decided to link to, and then, you know, there's a little blurb and it's got the picture. So I didn't, I didn't exclude the website, but I did not include the username, like the person who who pinned it. So I think because I feel like, yeah, the people do have certain expectations, like, or, or they, it's sort of more, it could, oh, it could be a lot more unconscious than that. Maybe not that they have certain expectations. But if you had them, if you ask them to stop and think about it, like, "Hey, would you like to see is published in a journal?" like, then they would think, "Yeah, I should give my permission for that, that to occur." So I don't think it's right to publish usernames, or if it was like something else really identifying like a picture of a person, I would definitely have second thoughts about that. But you know, that it was like a picture of [harmful misinformation]. And, and a little blurb about that, I felt I felt pretty comfortable with that. So, and the fact that the rest of the data is kind of a, what's the word? You know, de-personal, I mean, it's aggregated. So I don't feel like there's a violation there. And, you know, one person representing a .1% in misinformation, and there's no identifying information about what they posted, or any or who they are or anything like that. I feel pretty comfortable with that.</p>
  <p><b>Sara Mannheimer  </b>29:25</p>
  <p>Yeah. Tell me more about like, how you came to that conclusion? Like, you know, again, like did you talk to your research partner, or had you seen other Pinterest research projects, use similar ideas? Or did IRB weigh in or what sort of what sort of resources informed that that assessment that you made about informed consent?</p>
  <p><b>BSR07  </b>29:50</p>
  <p>I think it was a lot of it was the training that I received in the in the program. So like, we talked about this in our quantitative methods class. And again, I kind of I kind of have some qualms about just saying, oh, we're studying content, we're not studying people. But that was sort of that was probably my starting point. And then, you know, I, I'd have reflected from like, there was a time, there was a poster that I had been involved in. I was like, third or fourth author, and, and it had like a picture, it was, I think it was was Instagram that we talked about. And this was some time ago, this was like, you know, previous, prior to this Pinterest project. But I remember at the time, like, so I didn't prepare the poster.</p>
  <p>And I remember seeing the poster at the at our annual conference, and sort of thinking myself, like, maybe we do need to, we should ask permission, when we're using, like actual pictures of people that they posted on Instagram thinking, you know, not even thinking who is going to see it, besides their friends not expecting that some researchers somewhere would use it. So that kind of like influenced my thinking. But it didn't really apply to this, you know, that puts that informs my sort of general mindset about informed consent, but it didn't really apply to this project. And, and just regarding IRB, like, I knew that our institution was not going to, I mean, that was something that we were told very clearly. And in quantitative methods that like, for content analysis, you do not need IRB approval. So yeah, and then do that.</p>
  <p><b>Sara Mannheimer  </b>31:37</p>
  <p>Yeah. Okay. Sounds good. Okay, well, closely related is privacy and confidentiality. So you talked a little bit about this was like seeing the people's images on the poster. But for your project, and really, for any project, I guess, that you would use, like content from social media. Can you tell me about a time when you have considered issues of privacy? So I guess, protecting the data during the research. And then even afterwards, like if you consider the data sensitive in any way, because it has usernames, and then also thinking about that idea of allowing people's data to be out there, it sounds like you were pretty careful about it only giving the one example that was deidentified. But what other thoughts did you have about privacy when you're doing this research or similar research?</p>
  <p><b>BSR07  </b>32:35</p>
  <p>Um, I mean, although I didn't have or, or feel that I needed IRB approval I, I did just think about like, I-, what does IRB say about confidentiality, and, you know, the approach that I've been told is sufficient or is a good approach is basically that you are storing the data on a password protected computer. I backed things up to a external hard drive, and that's also password protected. And I know I don't, oh, and also, I shared some data with my co-author on Google Drive. And, and it was, you know, again, it was it was shared only with him. So I feel, I feel like those are sufficient steps to, to safeguard confidentiality and privacy. And I didn't have to say I didn't probably think too much beyond that. I mean, if someone raised an issue, if, if I were told that, you know, these are there's other steps I have to take, I would think about what else, you know whether I should be doing something more.</p>
  <p><b>Sara Mannheimer  </b>33:47</p>
  <p>Yeah, that's really interesting to hear that in your program they've presented it as so cut and dry, but then you like think further, and have additional concerns. Um, and so what did you think... Like, what do you think the participants -- the people who had pinned these pins -- What did you think their expectations of privacy were? Like do you think they would be shocked to learn that you're using the pins for this study? And do you think that would be okay with them?</p>
  <p><b>BSR07  </b>34:27</p>
  <p>Um, I mean, I think I think Pinterest is a pretty public platform where people expect, you know, I mean, I've, I've used it a little bit myself, and I'l  see, I mean, even given that I've hardly pinned anything from time to time, I'l  get a notification that like, you know, random person x that I've never heard of has shared my pin to a board. So I think that most users know that their that their pins are being viewed by and also pinned by other people and, and are being kind of spread around in that way. To be honest, like I don't, you know, probably most of them haven't thought about the possibility that their pins would be used for research. And I guess like, I feel like if you told them, you know, this, that their name is not going to be published, their username is not going to be published, their, you know, their image, image or other identifying information is not to be published, I feel like, most people would probably be pretty understanding of that. But I mean, I guess some people wil  probably feel a little weird that just like, who I've become this, become this, like data point in this big, or maybe, you know, oh, this person doesn't understand what I was saying here. Like, no, that's not misinformation, because, you know, they've, they've misconstrued what I was saying. So, like, I could definitely feel like people could have misgivings. So there's some concern there, but I don't know, I feel like it's outweighed by the fact that, you know, we're trying to document something that might be harmful and trying to help [society]. So yeah, feel like, feel like on balance, it's the it's an acceptable practice.</p>
  <p><b>Sara Mannheimer  </b>36:30</p>
  <p>Nice. Okay. Great. Al  right, we're on to our last question of intellectual property. Was there a time during the process of this research when you considered intellectual property concerns? It could be like social media Terms of Service or participant intellectual property.</p>
  <p> <b>BSR07  </b>36:52</p>
  <p>Um, you know, the, the time that that actually, like, came up for me, the most was when I was I was thinking about using Pinterest and [the scraping tool], again, for a slightly different project for a class.</p>
  <p>And, and [the scraping tool] just wasn't working with Pinterest anymore. And I thought to myself, maybe, you know, I sort of thought we were on the right side of the Terms of Service, but maybe Pinterest got this, the idea that like, "Oh, these researchers are using [the scraping tool] to do this scraping, and we don't like what they're finding and, or it is a violation of our terms of service." So, um, yeah, it kind of did give me a little bit of pause, then. I mean, I, I think it was sort of, I think it was acceptable use because I was, you know, I was just getting the URLs and then I literally had to go in manually and, and take all those. And also, I wasn't like, you know, I wasn't... It's not like I was republishing the raw data, I think I would have had more concerns and probably thought about this more deeply if I'd been sharing the raw data or putting it somewhere a lot of people could access. And I understand that, like, open science is a lot is a desirable thing. And a lot of ways, but like, yeah, there's just so much to think through and, and intellectual property is, is definitely one of those things. So I kind of felt like I didn't, I didn't have to examine that too closely. And I guess same thing when it comes to the users like, I don't know. Yeah, it's a diff-, that's a difficult question, because I feel like, it might not if you were publishing an entire pin. Does that... is that Fair Use because that is, you know, that one of the tests of Fair Use is usually like that you're only using a small proportion. And using an entire pin does seem like a little egregious in that way. And I guess, I don't know, again, it's just like, a kind of cost benefit, ethical analysis, where I was thinking like, well, you know, this one pin that I'm publishing is, is</p>
  <p>[giving harmful misinformation] and I don't feel a whole lot of ethical obligation or compunction to, to worry about, whether that's Fair Use, I don't know that that might be a flawed, ethical approach, but.</p>
  <p><b>Sara Mannheimer  </b>39:43</p>
  <p>It also sort of mirrors what you would do in a qualitative, you know, like, study where you interviewed people where you would put quotes and those are il ustrative and they help people understand the research. So I guess I would want to do that too. Yeah, yeah. Um, and did you consult with anyone about intellectual property with Pinterest research or use any guidelines or resources?</p>
  <p><b>BSR07  </b>40:10</p>
  <p>Um, you know, I think like the only time that question came up was probably when we were in the latter stages of the publication process, like I'm sure the journal had a question about, because they always do have those questions about intellectual property. And I wish I could, this is this is where it might be a bad thing that this was a couple of years ago. So I wish I could remember exactly what they asked about it. But I think that it sort of did cause me to, to just think and reflect for a little bit about like, okay, are we all right using this, this wil  just one pin. But there, there wasn't any time that I really like sought, sought advice or anything like that.</p>
  <p><b>Sara Mannheimer  </b>41:04</p>
  <p>Yeah. And I guess like, you're using just like what you've learned in school, what you've known from being an academic, what you've learned from previous publications that you've been part of, and all that's coming together to help you make these decisions?</p>
  <p><b>BSR07  </b>41:22</p>
  <p>Yeah, that's definitely yeah, that's true. I mean, yeah. The, like, you said: the PhD program, prior publication experience. Um, I mean, I may have had some discussions with my co-author about some of these issues, but I don't remember any of them being like, like, really a struggle or debate or anything like that, where we tussled with what to do.</p>
  <p> <b>Sara Mannheimer  </b>41:47</p>
  <p>Yeah. Okay. This is very interesting. Thank you so much. Are there any other issues or challenges that arose during your research that I haven't asked you about that you think would be of interest in terms of data curation and responsible use of social media data?</p>
  <p><b>BSR07  </b>42:09</p>
  <p>I'm trying to think if anything else came up? I don't think so. That's, I think that covers it pretty well. I mean, if I think of anything else should do you want me to drop you an email? And?</p>
  <p><b>Sara Mannheimer  </b>42:24</p>
  <p>Sure, like, if you want to, yeah.</p>
  <p><b>BSR07  </b>42:27</p>
  <p>Okay. Al  right.</p>
  <p><b>Sara Mannheimer  </b>42:30</p>
  <p>Yeah. Cool. Al  right. Wel , that is everything I have I okay. So thank you so, so much. I really appreciate you taking the time to talk to me.</p>
  <p><b>BSR07  </b>42:49</p>
  <p>Yes. My pleasure. Yeah. And this was really interesting. Thanks for inviting me to participate. And if there's anything I can do to help with your research, let me know.</p>
  <p> </p>


name: BSR_08
filename: Mannheimer_BSR08_Tra.pdf
created: 2024-07-11 14:35:13.225941
text_direction: LEFT_TO_RIGHT
contents:
  <p>BSR08_transcript_deidentified</p>

  <h2><b>SUMMARY KEYWORDS </b></h2>
  <p>research, data, people, twitter, researchers, tweets, context, social media, social media platform, analyzing, weibo, public health, individuals, publish, studies, users, questions, specific, instagram <b>SPEAKERS </b></p>
  <p>BSR08, Sara Mannheimer</p>
  <p><b>Sara Mannheimer  </b>00:03</p>
  <p>And so the summary of the research I sent this to you but is that so big social data, which I am cal ing it here, like social media, blogs, other online sources that people post, and archived qualitative data like interview transcripts, notebooks, diaries are similar. They're both qualitative data that people post online for one reason. And then they're used for different reasons for research or a new research. But I found that their respective communities of practice are under connected so big social researchers tend not to talk as much with qualitative researchers or people who do qualitative research. And so this research is trying to investigate how librarians can use data curation practices from each of these communities to bring them closer together for mutual benefit. To help qualitative researchers scale up their research by reusing data, and to help social media researchers be more responsible. And or just do more with social media data. Yeah. So any questions before we begin?</p>
  <p><b>BSR08  </b>01:49</p>
  <p>Yeah, I've read your document. And I don't have specific questions. I may have other questions as we previously communicated after you interviewed me about more about your, for me to learn about your research and others, but I think I'l  let you finish interviewing me first. And I hope I'm able to answer you questions from a specific perspective, which is [some details redacted], epidemiology, and digital health research. [Additional details redacted]. Digital health is actually a very, very broad domain. And it is a relatively new discipline, and when I say new—probably in the past 10 to 15 years. But if you consider public health as a discipline of over 200 years, this is real y new. But digital health in itself contains a variety of components. And it ranges from telemedicine, to mobile health, to health communication analysis, using social media data, to using social media and other digital device data to predict epidemics, to using digital devices to help patients monitor their health and so on. So, the type of research that I have previously done primarily is analysis of social media data, initial y using that as some of the variables that could help forecasting in epidemics. But it turns out, either perhaps because I'm not so good at forecasting or perhaps these, these data may not be a very good information input for for epidemic forecasting, and and my research took a turn, into the domain of digital health communication research. I also find that the majority of health communication researchers are not where they are, they're not comfortable with or they do not appreciate the type of more big data type of research in the communication realm. While there is a tiny fraction of communication researchers who do this type of big data analysis, they tend to form their own small community. It turns out that as I realized, because my training is not really in communication, but in epidemiology, so my concern is not really answering questions that communication scholars ask in terms of the types of theories and so on.</p>
  <p>And so that is why the type of research really doesn't really fit into the communication studies or journalism studies. So the reason why I spent five minutes talking about this is to... now you know the...</p>
  <p>it sets the background, sets the scene against which my answers to your following questions should be understood with that background.</p>
  <p> <b>Sara Mannheimer  </b>05:12</p>
  <p>Great. That was actually my first question was to tell me about the type of research you do and the type of data that you produce or use. So I feel like that really covers that. So is most of the data that you collect now from social media? Or do you do surveys and other types of data collection as well, for your epidemiology work?</p>
  <p><b>BSR08  </b>05:33</p>
  <p>I may do surveys but I don't do much. This, I mean, even surveys, not really epidemiology when we talk of survey, usually, when you're talking talking about behavioral health science, or community health.</p>
  <p>Different names for the same thing like basically the psychological, sociological or economical variables that that are predictors of certain types of behavior, for example, why somebody chooses to take the vaccine or not take the vaccine against COVID-19. But the type of... back to what I've said, the type of research that previous, previously that I've done, basically, prior to COVID-19. I wil  say it's not primarily quantitative type of research of social media, they are primarily Twitter. But I have done research in Weibo. I've also done a little bit of research in Instagram, and Pinterest. However, I, the type of research that I have done, I wil  say, is semi, semi quantitative, i.e., they're not very, they're not the type of thing that you hear from media where you have supercomputers doing deep learning and discover something that you really, we can't really consider but a computer algorithm generate that. I mean, I'm not a computer scientist on that. But at the same time, I'm not I was not doing the type of traditional qualitative research, where people are wanting focus groups, or this time one on one interviews, providing a lot of context to the specific tasks or specific documents or specific social media posts that they generate. Yeah, so meanwhile, right now, I'm more stepping back a little bit becoming sometimes why some, some, some form of an analytical support to communication scholars where they'd has to do more qualitative type of research. Yeah, so I don't know if I fully answered your question. But maybe, if I, you can let me elaborate, if you depending on what...</p>
  <p><b>Sara Mannheimer  </b>07:58</p>
  <p>I think... I think that's good. I'm just looking for kind of a background as you said a foundation to help understand where the rest of the interview goes. So the next thing we want to do to establish foundation is to talk about a specific example, I'd asked you to identify a recent time when you collected big social data for your research, or use social data that was collected or shared by someone else. And this example, we don't have to stick to this example, alone the entire time, but it's helpful to have the specific time in mind as we walk through these questions. I'm going to be answering or asking questions about six key issues that I identified through a lit review, context, data quality and trustworthiness, data comparability, consent, privacy and intellectual property. And so we'l  talk through any issues or challenges you encountered in those areas with this specific project. So do you have a project in mind that you want to use for today's interview?</p>
  <p><b>BSR08  </b>09:07</p>
  <p>Well, any one of them. But I could choose. In fact, I want to, I want to talk about I want to use three different examples to il ustrate. The one is a paper that I previously published in [an academic journal]</p>
  <p>about [infectious disease] misinformation. Okay, that was a comparison between Weibo data and Twitter data.</p>
  <p><b>Sara Mannheimer  </b>09:41</p>
  <p>What's the first type of data We, Weibo?</p>
  <p><b>BSR08  </b>09:43</p>
  <p>Weibo, Weibo. Yeah, Weibo, Weibo. Yes, that is the, that is China's equivalent to Twitter, because Twitter is banned in China as you know, and so W E I B O. Okay, so Weibo literally means microblog.</p>
  <p>Yeah, but but Weibo in itself is no longer is no longer something trendy in China. Most Chinese switch to WeChat. Anyway, so so it was in a historical context. But that is number one. And other examples. I think I have a study about using Instagram and Pinterest. That is more qualitative. I don't know if I have the link to that. Or you can find it on my website, I think.</p>
  <p><b>Sara Mannheimer  </b>10:42</p>
  <p>Yeah.</p>
  <p><b>BSR08  </b>10:42</p>
  <p>Yeah. So so that but I can briefly mentioned the challenge with regard to collecting information for Instagram, Pinterest, or any other thing other than Twitter. And then the third one is an ongoing study that my colleagues and I are doing using the Twitter data that Twitter released in 2019. Basically, they are tweets that were posted by accounts that were banned by Twitter. Okay, those accounts were considered by Twitter after the internal investigations as basically information operatives, by certain governments who are deliberately propagating this information or other whatever, whatever the names we call it, in order to achieve certain types of political goal, either internally or externally, i.e., involving international relations. So that a different context because it was questions about like, the context of privacy and that kind of thing. So I think this my using these three different examples can can give them more comprehensive answers to a question you asked.</p>
  <p><b>Sara Mannheimer  </b>12:06</p>
  <p>Okay, nice. The sounds great. What data collection method do you use for each of these types of research?</p>
  <p><b>BSR08  </b>12:17</p>
  <p>So for for Weibo, data, I think we were I've got a so so okay, so I collaborated with colleagues who had who previously used an API to call out Weibo data. They, my colleagues, my collaborators had, basically set up cohorts to follow certain publicly available Weibo accounts and to collate that data and then we were using that. So so it was an API. And for Twitter, I, if I remember correctly, it was also API, I think it was 1%. You could do either as a 1% API. So it was supposedly a random sample. But of the entire Twittersphere, even though we don't know exactly if it is really, truly random. Or you could use a certain keywords in order to search them. But there's also limits to how far you can get without paying. I have other studies that actually we have pay and buy a bigger share, i.e., for example, 10% stream or other for Twitter data. For the other studies that I mentioned, I think, for Instagram and Pinterest, I think Instagram, we were using API before they changed the user agreement. I think, after a certain—and I forgot at what time, Instagram changed the agreement, and severely limit the volumes of information that a researcher can download. So it basically make it extremely difficult to analyze. And I think for Pinterest, I think I had my students manually, manually screen for screen grab those information. And then the third study I mentioned for for the more recent one, the entire data set was released by Twitter as long as we provide a bonafide academic email address and our information then they wil  allow us to download the information for research.</p>
  <p><b>Sara Mannheimer  </b>14:41</p>
  <p>Okay. Great. And do you have is this part of a grant funded project, for instance, that required like a data management plan? Do you have a specific plan for how you handle the data?</p>
  <p><b>BSR08  </b>14:54</p>
  <p>For the studies that I have mentioned, they were not funded. Yeah, so so majority of my research are not grant funded.</p>
  <p><b>Sara Mannheimer  </b>15:04</p>
  <p>And do, wil  you plan to release the data you collect it for any of these three projects? Did you publish it as supplementary material or published it anywhere?</p>
  <p><b>BSR08  </b>15:17</p>
  <p>I did not. I do not. The major reason is that, especially, I mean, studying for the last example, I mean, the Twitter data is already released by Twitter for the public, for any researchers in the world, publicly available. For the other one, I think for, for Twitter data, whether we pay for it or we download it within the 1% stream that we're free to do. I don't think that we are allowed to upload those Tweets somewhere else, I think I think Twitter wil  require individuals researcher to directly get in touch with them and get permission to get hold of the Twitter data.</p>
  <p><b>Sara Mannheimer  </b>16:09</p>
  <p>What I've seen people do is they are only allowed to publish the tweet ID. And then they have these tools that can happen is called Hydrate the tweet ID and bring in all the other metadata that was there.</p>
  <p>But then it means that if you hydrate the datasets, no tweets from people who have deleted their accounts or have deleted specific tweets, those tweets don't populate, you know. So that's what I've seen people do for Twitter. Yeah, it's not ideal.</p>
  <p><b>BSR08  </b>16:36</p>
  <p>Yeah, sure. I mean, I did share that. I think one time, I mean, last year, somebody at another major university, I think someone, some students doing student projects with their professor, we request, to manually copy the data, of the Weibo and Twitter data for my [infectious disease] misinformation study.</p>
  <p>Because they want to, repeat the study for something similar, but want to compare with what I've done.</p>
  <p>And I sort of gave the deidentified a version of the manually culled tweets, and manually cul ed Weibo to the students in the other university.</p>
  <p><b>Sara Mannheimer  </b>17:20</p>
  <p>Oh, that's so cool.</p>
  <p><b>BSR08  </b>17:21</p>
  <p>So but that was between them and me through emails and and blah, blah, blah, I am not... So in essence, we are not releasing it into a public sphere, like, it is not in the certain database.</p>
  <p><b>Sara Mannheimer  </b>17:37</p>
  <p>Right. Yeah. Did you see what they ended up doing with that data? Was it just a project? Or did they publish?</p>
  <p><b>BSR08  </b>17:45</p>
  <p>Is a good idea? Maybe I should ask them. They did not contact me after that semester. So I don't.</p>
  <p><b>Sara Mannheimer  </b>17:52</p>
  <p>That's so cool. Okay, let's move on to the first issue, which is context. So I have a little quote, to help understand where I'm coming from. And when I talk about context. So Halavais suggests that "when we collect data from social media platforms, just like when we collect data in traditional spaces, context matters. But the context of the social media posts could be absent or difficult to understand, since these</p>
  <p>posts are, by nature, short pieces of text, or images or videos that are taken from a larger context of someone's personal or public life. And then the more data you have, the more the out of context effect can be compounded." So can you tel  me about a time if there were any during the process of one of these or a few of these examples we have in mind when you considered the issue of maintaining or understanding the data's context?</p>
  <p><b>BSR08  </b>18:53</p>
  <p>So to take the [infectious disease misinformation study] as an example, I think that there's... we are only looking at this very specific times where the [the outbreak was affecting both the U.S. and China]</p>
  <p>at the same time. So the context, are you referring to a specific outbreak at that time, is relatively clearly defined. And for the type of research that I've done previously, both the first example and the second example. Each of these are primarily are directly related to certain ongoing epidemics of emerging infectious diseases, and therefore, the contexts are relatively clearly defined. However, should digital health researchers choose to analyze data patterns to have conditions that are more prevalent... Take for example, I have another paper that was about [another infectious disease, centered around an event]. So my students looking at those tweets that were around the time of [the event]. You can say the context is clear in terms of we're looking at the conversations primarily promoted [around the time of the event]. But at the same time, the local context from which these individuals are advocating from, may not be clear to us as researchers, because the context of [the infectious disease] and the stigma associated with [the disease] in let's say, [example country] might be very different from from [a second example country]. And and given the fact that many individuals who use social media do not release or disclose their locations, we may not or we do not know where they are from, and therefore, the local context cannot be discerned.</p>
  <p><b>Sara Mannheimer  </b>21:06</p>
  <p>So what can... does that concern you? Did you have like, did you talk about it in your paper? Or were there other things that you did to try to support more of this data collection, in context? Were you concerned that your results would that you wouldn't have the full picture? And if so, how did you talk about that?</p>
  <p><b>BSR08  </b>21:30</p>
  <p>So I think context can be interpreted in multiple ways. Okay, so first of all, think about the geographical location. So I think it is, within computer science, a lot of people are talking about how to input missing data, or missing locations of Twitter users. And actually [I've seen entire research studies exploring new ways] of inputting the locations of Twitter users. So this is, could be very specific, very technical aspects of context. But then you can also have a different type of context, i.e., the context of the conversation. So somebody can be engaging in conversations about specific diseases, be they Ebola or Zika, or HIV or currently COVID-19. And when we talk about a context, you could talk about a local context, for example, I mean, the United States are more concerned about the response to COVID-19</p>
  <p>by President Trump and President Biden, while someone else living in the United Kingdom wil  have a very different political environment, very different public policy, and therefore the entire conversations about COVID-19 on Twitter wil  be very different. But it could also be a different context, when we talk about thing about people's who debate about the safety and advocacy of COVID-19 vaccine on Twitter, as such debates can both be localized, but also international, because people are responding to the same vaccine, people are reacting to this the news of the same vaccine being introduced across the world. And so people across national boundaries across political jurisdiction, are engaging into a conversation about the same thing. And this in itself is a context for our conversation. So back to your question back to the examples that I have. I think that in the papers that I did my research, we do explain the challenges of locating these individuals, and we do not know where they are a lot of the</p>
  <p>time, some of them we do have a geographical location, but many of them we do not. And therefore, it is very difficult to define whatever observations we have to a specific geopolitical jurisdiction, and therefore the health slash political slash economic context that these individuals may find themselves in.</p>
  <p><b>Sara Mannheimer  </b>23:58</p>
  <p>Yeah. And so did you talk about this issue with anyone on your research team? Was it something that like you sort of as you were designing the research talked through and considered? Or are you thinking that? Yeah, tell me more.</p>
  <p><b>BSR08  </b>24:17</p>
  <p>Well. Yes, we certainly do. And to use the third example that I mentioned: the Twitter data that have been released by Twitter, because it was really about different types of governments who are doing this information or slash whatever information operatives, what they were doing... the socio political economical context becomes very important. And so you know, either we are writing, certainly my communication scholars, colleagues, who are from different geographical political background, we certainly have to discuss the specific context in order to understand what the specific government alleged by Twitter who may or may not be doing information operatives? And if they are, if they were, what were their purpose what were their goals. And did they achieve anything? If so, what are they?</p>
  <p><b>Sara Mannheimer  </b>25:14</p>
  <p>Yeah. Okay, great. Let's move on to data quality. Can you tell me a time during the process of one of the one or more of these research projects when you considered the issue of data quality? So for example, like missing data from the Twitter API, or but or bias among the Twitter users, or other issues of quality or trustworthiness of the data.</p>
  <p><b>BSR08  </b>25:43</p>
  <p>Okay, so from a public health research point of view, especial y from someone with an epidemiology background, a major concern that we have is whether or not the data are representative of the populations that we are interested in. And first of all, we know that people who use Twitter or any other social media, they are only a subset of the population, okay, they may not be representative of the general population. Now, for social media, such as YouTube, or Facebook, who may have more than half of Americans using them, they may be relatively representative in the context of the United States.</p>
  <p>But for social media like Twitter, which only a quarter of Americans are using it, it is already not representative among Americans. And when we're talking about geopolitical jurisdictions, in other parts of the world, the coverage, or the usage, in terms of the proportion of the population being active users of social media, they are not representative of the general population. So therefore, public health researchers usually have to indicate that whatever results they found can only be representative, among the... representative of the users who are social media active in their respective geopolitical jurisdiction, and not the general population. And that is why the it is extremely difficult to use social media data as an input for epidemic forecasting, because epidemics involve the entire population most of the time. While if you are talking about communication research, maybe the research question is different. You are interested in what people are talking about on social media such as Twitter, among those users who also are social media active. And you are not talking about the entire general population. And so that is why, in terms of epidemiology, I think the utility of social media data may not be as good as some advocates would like to admit. While for communication research, I think it is less a problem.</p>
  <p><b>Sara Mannheimer  </b>28:02</p>
  <p>Yeah. So you basically only would ask certain research questions of Twitter... is what you're saying?</p>
  <p>You sort of use it for research questions that you think it wil  be more useful for?</p>
  <p><b>BSR08  </b>28:16</p>
  <p>Yeah, and that applies to any social media.</p>
  <p><b>Sara Mannheimer  </b>28:19</p>
  <p>Yeah. Again, thinking about, like how you work through, you know, like, you've come to all these conclusions about how to deal with missing data, or representative may miss of the data. How did you come to these conclusions, like did you consult resources or talk to your colleagues? Read other articles that people have been writing? What's been your process?</p>
  <p><b>BSR08  </b>28:48</p>
  <p>Well certainly literature, literature review is certainly a big part of that. And I think we are very aware of the bias involved in social media data. I mean, it certainly became a major issue after some of my early papers. And I think for many individuals who more recently ventured into this field of trying to analyze the submitted data as if it represents somehow, an epidemic signal and something like that. It has been I think it is, it could have been an easy pitfall for many, until the time they realize that this is not really representative of population. So back to your question is, how would I be aware of this potential pitfall?</p>
  <p>I was a both literature reviews but also, reviewers' critique. I think I have submitted enough numbers of my manuscript and gotten rejected enough number of times before they get published. And so I think it has certainly gradually become a consensus among public health researchers, in fact, many public health researchers, especially people of the older generation, they are in general, very skeptical of this type of research.</p>
  <p><b>Sara Mannheimer  </b>30:16</p>
  <p>Great. And so did you....Do you write a section in most of your papers that talks about this? Yeah.</p>
  <p><b>BSR08  </b>30:25</p>
  <p>Limitations. Yes. Yeah. So certainly one, or more than one paragraphs of limitations and discussion, yes.</p>
  <p><b>Sara Mannheimer  </b>30:35</p>
  <p>Great. Al  right. Let's move to data comparability. So during any of these examples, it seems like at least with that Weibo and Twitter one, did you combined multiple big social datasets to try to do your research?</p>
  <p><b>BSR08  </b>30:50</p>
  <p>So for the two examples that I chose to talk about -- both the one with Weibo and Twitter and the other one with Instagram and Pinterest, certainly they are being presented within the same paper. But I'm not saying that we combined them, but we certainly compared them.</p>
  <p><b>Sara Mannheimer  </b>31:06</p>
  <p>Okay.</p>
  <p><b>BSR08  </b>31:07</p>
  <p>Yeah.</p>
  <p><b>Sara Mannheimer  </b>31:08</p>
  <p>And what... Let's see, what strategies did you use to compare? Like, did you did you need to combine the data into one data set to do your analysis? Or did you do two separate analyses and then compare those two analyses?</p>
  <p><b>BSR08  </b>31:32</p>
  <p>I think in... I think in both cases, we we, we would have certain types of framework, that that that generates the codebook that allows us to manually code the sampled data, so that we can... because in both cases, we are comparing content. And so we wil  need to have something to, something to compare with. So so basically, we need to standardize categories of contents to which we coded the social media posts. And then we can compare for example, whether or not this information about [the infectious disease] is is higher in one data sets and the other on. There's no difference.</p>
  <p><b>Sara Mannheimer  </b>32:28</p>
  <p>Okay, so the way that you... you used qualitative methods to analyze the data? Did you like, read through each of the tweets and coded them by hand?</p>
  <p><b>BSR08  </b>32:41</p>
  <p>Yes, so for some of my studies, they were entirely qualitative manual coding. For some of these other studies we have used -- not the examples I chose to present here -- But for some other studies, we did use a Latent Dirichlet Al ocation or some other type of topic models to allow basically the computers to identify potential latent, i.e., hidden topics. But then we stil  use manual coding to code the topics into humanly understandable categories of contents.</p>
  <p><b>Sara Mannheimer  </b>33:14</p>
  <p>Got it. Okay, that's helpful. Great, did you encounter any other challenges? When you are trying to combine datasets in these studies?</p>
  <p><b>BSR08  </b>33:27</p>
  <p>We'l  take some examples, if you have... if you're doing comparisons involving data sets of different languages. So for example, I have another study that we basically compared the data between... on on the same topic, but it uses a different, different language. So we have different codebooks. Okay, so one of my most interesting studies, in fact, was comparing the same topic, but across different languages, tweets on Twitter. So even though it's the same platform, in fact, there are users of different communities who speak different languages. You would either need a coder who understands those languages, so you would need a team helping you. Or you would need to use technology, such as Google Translate API or Bing Translate or any other platform that may be free or they may charge charge money. But but either way, you wil  need help from human or technology to assist.</p>
  <p><b>Sara Mannheimer  </b>34:34</p>
  <p>Yeah, great. Okay, let's move to informed consent. This is our fourth topic. So can you tell me about a time if any, during the process of these research projects when you considered informed consent?</p>
  <p><b>BSR08  </b>34:54</p>
  <p>So technically, I don't have participants in the sense, we certainly have IRB approval, but they usually have certain types of exemptions. Because the type of data that we get are publicly available data. So somebody voluntarily consented to post those information online to let the world see it. And so we do not. So so. So IRB committed... did not consider that these studies require actually an informed consent, because technically, there are no participants.</p>
  <p> </p>
  <p><b>Sara Mannheimer  </b>35:33</p>
  <p>When you're talking about sort of public health topics, has it become like, did you ever have any concerns about using the tweets? Or did it, was it pretty clear? Did it feel clear that if the IRB approved it and it's publicly available, these tweets are okay to use?</p>
  <p><b>BSR08  </b>35:55</p>
  <p>I think so but of course, we I mean, we usually would not, in the paper publish, the Twitter handles, or the names of individuals, except for organizations. Let's say we can say the CDC or the the US</p>
  <p>government tweets about something. Or they are really prominent celebrities, which they can't I mean, claim, privacy in that context.</p>
  <p><b>Sara Mannheimer  </b>36:29</p>
  <p>Yeah. And did you feel...? So you're kind of saying that you figured that individual people wouldn't necessarily expect their tweet to appear in a research paper, but bigger organizations and more public figures? You felt that those people would be okay with it? Is that right?</p>
  <p><b>BSR08  </b>36:55</p>
  <p>It is just like, if you're reading a news platform by major media organizations, they would also, quote tweet, for example, and it could be unknown people, when most of the time they may say, oh, President, former President Biden's tweets say about something, or whether it is Bil  Gates or the Gates Foundation's tweets about the importance of vaccines, for example, something like that. So I do not think that... I mean, so, so basically, for government or other major organizations, or celebrities, I mean, all politicians, I mean, they afford their publicly available social media posts, I do not think that privacy would be an issue. But but when we're talking about, there's some random people on the streets, and I mean, we may mention that content, but we usually wil  not write down we wil  not write their Twitter handles into the paper.</p>
  <p><b>Sara Mannheimer  </b>38:14</p>
  <p>Yeah. Okay, this sounds good.</p>
  <p><b>BSR08  </b>38:19</p>
  <p>The data we use are publicly available data. So anyone with internet access, technically, can join to it and a technically can hand weight any of these trades. So, so we are not analyzing any post that were made private by the users.</p>
  <p><b>Sara Mannheimer  </b>38:43</p>
  <p>Okay. Nice. Okay, yeah. So the next topic is really closely related about privacy and confidentiality.</p>
  <p>Was there a time during the process when you considered issues of privacy? So you said you didn't use any tweets that weren't publicly available? What other considerations did you have regarding privacy of the users of Twitter or Instagram or Pinterest in some of this research that you did?</p>
  <p><b>BSR08  </b>39:17</p>
  <p>I think I've already sort of explained that. I mean, yeah. So we, we, we analyze publicly available social media post, and so I don't see major uses of privacy. Yeah. And just like just like people, analyzing newspapers, I mean, we don't really have privacy.</p>
  <p><b>Sara Mannheimer  </b>39:40</p>
  <p>Yeah. Yeah. So you're viewing the tweets as like, public websites that you're just scraping. Awesome.</p>
  <p>Al  right. Wel , this is our last topic, intellectual property... Can you tell me about a time during your</p>
  <p>research process when you considered intellectual property concerns. So could be like social media platform Terms of Service, or participant intellectual property, or your own intellectual property as a researcher?</p>
  <p><b>BSR08  </b>40:15</p>
  <p>No, I didn't. I mean, technically as university researchers, whatever products I thinks we generate, I mean, the the... it goes to the universities. But I didn't I did not invent anything that can qualify as an IP, a patent. So yeah, no.</p>
  <p><b>Sara Mannheimer  </b>40:33</p>
  <p>Yeah. Did you read the Terms of Service for like, it sounds like you did for Instagram and Pinterest and Twitter, and then take those into consideration as you were collecting the data? And did you try and meet the terms of service when you did?</p>
  <p><b>BSR08  </b>40:48</p>
  <p>The paper that we published, the data were collected before Instagram changed their terms of service.</p>
  <p>So right now, I mean, after that, it really... after that, it is real y impossible to do any meaningful research. And I do not see a way of getting around it. And so I did not do anything with Instagram after that. I did write a review paper with my students on... work about basical y those image-based social media. But at the time, at the time, when I wrote that review paper, I think, if I remember correctly, in the review paper, actually saying that for the data that we as a team reviewed, that involve Instagram, they were done before Instagram changed their Terms of Service. And so those published, public health related social media research that involves Instagram, actually, they cannot be repeated in the future. Yeah, and I do not think... I do not find a solution to that. Unless, okay, individuals, say you've managed to recruit 500 participants, and then they consent, and then you we don't even download the Instagram photos. That's a different story.</p>
  <p><b>Sara Mannheimer  </b>42:20</p>
  <p>Okay, so Instagram has said that there is no like, no more just like automated downloading, but you'd have to...</p>
  <p><b>BSR08  </b>42:27</p>
  <p>Have they have a cap, they have a cap, which means that it is very difficult to do big data with and, and you want things to be representative. So that's the challenge. Right? So that is a major challenge in public health related social media research. Today, with the exceptions of those that that recruit participants in the first place and ask them for consent, for the researchers to read and download their social media posts... with the exception of those type of studies, most type of studies that the type of study that I have done before primary secondary data analysis, type of big data research, that do not involve directly recruiting participants for their concerns. Those type of research are heavily tilted towards Twitter research, because it is relatively easy. And Twitter's basically people consent to publish their, their own writing to the world, right? Because like things like Facebook, people use it to communicate with friends that they know in real life, while for Twitter, they are pretty much a platform to to say something to the entire world. The audiences are highly different. So because of that, so Twitter doesn't really have that much privacy issues as compared to getting data from Facebook or any other social media platform. And therefore you find 1000s of studies on Twitter. As Twitter data analysis, I'm talking about public health related one, I don't know anything not related to public health, but public health related social media research, primarily are Twitter data analyses. While we know that Twitter uses only a -- It's not very small, but it is not even half of social media users in the world. And therefore, it is not representative. Especially for younger people. They're using Instagram, using Snapchat, using</p>
  <p>TikTok. While the majority of us who are professors are stil  analyzing Twitter, because because we grew up with that. I mean, this is a social media platform of our generation, but they are not a social media platform of my students.</p>
  <p><b>Sara Mannheimer  </b>42:45</p>
  <p>How do you like as a digital researcher, digital health researcher... What are your thoughts moving forward? Like, what do you think you'l  do in the future? Do you think these Twitter studies are stil going to be relevant? Or what are your plans for sort of addressing that, in your future research.</p>
  <p><b>BSR08  </b>45:27</p>
  <p>I think that they, they are, they remain relevant, as long as a platform like Twitter or Facebook, has bil ions of users using them. We just have to be aware of its limitation. So so it may be a little bit cynical, or a little bit... it's half true and half a joke, that eventually, Facebook research and Twitter research is like doing research with older adults.</p>
  <p><b>Sara Mannheimer  </b>46:04</p>
  <p>Right.</p>
  <p><b>BSR08  </b>46:05</p>
  <p>Conducting health related research with older adults. Just like if we are doing research, about diabetes, hypertension, that sort of thing, because we are reaching that population group. But if we are thinking about doing, let's say, sexual health education, let's say substance abuse issues among young people, analyzing Facebook and Twitter would not help you with your research goal. And so I'm sure there are also researchers who are doing research with things like Instagram, Snapchat, TikTok, and so on, but I'm not so familiar with it. And I haven't ventured really into that substantially.</p>
  <p><b>Sara Mannheimer  </b>47:00</p>
  <p>Does Twitter out of curiosity, does Twitter make available demographic information about its users? Or is that available anywhere?</p>
  <p><b>BSR08  </b>47:09</p>
  <p>I don't think so. Most individuals do not... Most individuals who use Twitter, so, uh, the majority of them, do not disclose much of their own information. So technical y, yes, they could. They could release their own information on their Twitter profiles, but the majority of them do not. And therefore, those who do is a tiny minority. And it is very likely they are not representative. They're not representative even of the Twitter users population, let alone the general population.</p>
  <p><b>Sara Mannheimer  </b>47:42</p>
  <p>Right. Okay, cool. Al  right. So my last question is, are there any other issues or chal enges that arose during these examples, these research projects that I haven't asked you about?</p>
  <p><b>BSR08  </b>47:56</p>
  <p>I think that is a major issue here with regard to protecting privacy of users, which is very important as we are also social media users. And we, as users, we want privacy. I mean, if, if this issue has not been tackled, we may not choose to use social media platforms at all. But from the researcher point of view, this creates a hurdle that many researchers actual y cannot overcome. And therefore, this is precisely the issue that makes public health related social media research heavily tilted towards Twitter. That means that while we as public health researchers understand the importance of the newer social media platforms as they are used by young people, we are not devoting as much time and resources into</p>
  <p>analyzing them, and understanding the issues of the young people that deserve our attention. And I have not yet seen a major revolution in terms of how we as public health researchers are going to handle these issues. Yeah, because... because the way to go forward to overcome the issue of privacy actually would be to recruit a large enough sample of individuals, ask them for their consent, and then they have to surrender their social media data to the researchers.</p>
  <p><b>Sara Mannheimer  </b>49:39</p>
  <p>Yeah. Okay, I was gonna ask if you have heard of people actually doing that. Yeah.</p>
  <p><b>BSR08  </b>49:48</p>
  <p>Essentially, that could be... but you wil  need a big grant. And usually, it wil  be a certain subset of the population with whom you build a relationship. Let's say, let's say individuals who may get a grant from NIH to do research about delivering health information to a certain group of patients, maybe, let's say, HIV positive individuals or individuals with hypertension. And you're either using a social media platform as a health education platform... a channel that these people consent to participate, or you provide them with certain digital device that you promised them, this wil  help them manage their hypertension, and they sign their consent forms. They know what benefits they could get in return, and therefore they are wil ing to participate and provide you additional information on top of the demographic information that you collect -- maybe their social media information -- to assist with the research. But now we'l need NIH or other types of research grants that many [researchers] in tier two institutions or tier four institutions do not have. In fact the reality is, why so many people, including myself, are analyzing social media data in the first place, is because we do not have a big grant to recruit 1000 people.</p>
  <p><b>Sara Mannheimer  </b>51:12</p>
  <p>Right.</p>
  <p><b>BSR08  </b>51:13</p>
  <p>For research, right? That is why the convenience of getting secondary data is that yeah, it is, even if we pay for it is stil  not too expensive. We have expertise analyzing them. And they have okay, we can get papers published and satisfy our own curiosity. And yet meet the needs of the University and people move from professor to associate professor.</p>
  <p><b>Sara Mannheimer  </b>51:38</p>
  <p>Yeah, that's such a good point. And I feel like that's part of the argument for sharing qualitative data, too, is like, well, it's, you know, it's time consuming. And I have, you know, I had to ask you for this huge favor to spend an hour of your time talking to me, you know, to gather qualitative data. And so if we can publish more than there wil  be just like with social media, it wil  be like there is this, you know, this true treasure trove of information that then researchers can use to ask different questions. So, yeah, that's such a good point.</p>
  <p><b>BSR08  </b>52:16</p>
  <p>And also another question was, the pace of academic research is slow, relative to the development of social media platforms. So take the environment of social media in China as an example. [A few details redacted regarding BSR08's research experience]. Because my collaborator collected those Weibo data sets for other research topic. And I simply... we use them for health, looking at health keywords and health-related questions. By the time some of my early papers got published. I think there's a cohort of researchers of similar age group who picked up the ideas, and they followed my ideas and repeated that for many other health topics. In the following two, three years, I moved on to analyzing Twitter data. However, by the time many of these research got published, and get the attention of the</p>
  <p>non Chinese speaking, scientific community, let's say, the western scientific community, the use of Weibo, among Chinese speaking individuals has significantly declined. Or they they remain users, but they become inactive. That was because of the Chinese government's policy of suppressing voices of dissent within the social media platform, and also with the emergence of WeChat, which is pretty much China's equivalent of WhatsApp, with additional functionalities. And the Chinese public basical y switched over to WeChat. And likewise, by the time public health researchers realize the importance of WeChat and start to find ways to analyze WeChat -- which is actually very difficult -- the, the WeChat sphere, so to speak, in China, and the Chinese speaking individuals around the world has also evolved.</p>
  <p>Primarily because of the major suppression of voices of dissent on WeChat by the Chinese government in the recent two years. So the social media landscape changes a lot, very quickly, but the cycle of academic productivity has longer wavelengths so to speak.</p>
  <p><b>Sara Mannheimer  </b>55:23</p>
  <p>Yeah. That's such a good point. Yeah, I mean, I'm finding that even in the three years that I've been working on my PhD, it's like each year that we you know, the Twitter Terms of Service change, or you know, Instagram's Terms of Service, or yeah, people are moving platforms. I've definitely experienced that, too.</p>
  <p><b>BSR08  </b>55:42</p>
  <p>So back to your question, I actually want to say one more comment is that I see your I see your perspective, as, as a librarian and Data Manager, how you could facilitate sharing of data between researchers and prevent duplicating effort and so on. But by the time a data sets can be shared, it's old.</p>
  <p><b>Sara Mannheimer  </b>56:02</p>
  <p>Yeah.</p>
  <p><b>BSR08  </b>56:02</p>
  <p>So it's a historical data set. So so for the fast-changing field that I am in, let's say we are researchers of emerging infectious diseases. We are all analyzing COVID related data. Ebola of 2014 sounds like ages ago. Nobody's talking about Ebola right now. Nobody's talking about Zika right now. So yeah, yes.</p>
  <p>Even if I am available to or I am capable of sharing data, that was six years ago. Who is going to use it?</p>
  <p>I mean, maybe a medical historian, but not the active researchers who are chasing the next epidemic.</p>
  <p><b>Sara Mannheimer  </b>56:43</p>
  <p>Mm hmm. That's a really good point. Yeah. I mean, I think, you know, you could use it for longitudinal studies, or comparative studies where you look at how, you know, Zika spread or, you know, in comparison to a new similar disease or something like that. But you're right. It's like, if... thinking about science as this constant pursuit of the cutting edge, that's problematic. Yeah. Cool. Al  right. Well, that's all I have for you.</p>
  <p> </p>


name: BSR_09
filename: Mannheimer_BSR09_Tra.pdf
created: 2024-07-11 14:35:23.167774
text_direction: LEFT_TO_RIGHT
contents:
  <p> BSR09_transcript_deidentified</p>
  <h2><b>SUMMARY KEYWORDS </b></h2>
  <p>tweets, data, research, twitter, collected, people, infrastructure, privacy, intellectual property, post, issues, users, context, thinking, paper, news articles, area, api, question, information <b>SPEAKERS </b></p>
  <p>BSR09, Sara Mannheimer</p>
  <p><b>Sara Mannheimer  </b>00:03</p>
  <p>The research basically is about connecting two communities of practice that I've found to be under-connected, which are people who do research with big data, big social media data, which I'm calling big social data in my project, and then people who reuse or share their qualitative data. So I'm sort of envisioning social media data as a type of qualitative data. It's text, images, videos that's shared online.</p>
  <p>And then when you use it for research, it's kind of like you're reusing it, you're putting it to a new type of use than what it was originally intended for. So my hope is that by talking to researchers who have done big, big data research, and by talking to qualitative researchers, and talking to data curators like myself, I'l  be able to develop a set of strategies that data curators and librarians can use to support responsible research in both of these areas. So that's why I'm talking to you today. I think one of my other research participants recommended you, saying that you were working with this type of data and that you might be good to talk to so. So yeah, the interview is going to be structured around these six issues that I've identified, which are context, data quality, and trustworthiness, data comparability, consent, privacy and intellectual property. So I'l  ask six questions. I've got a little intro and conclusion question. It should take about an hour, maybe less, sometimes 45 minutes. Yeah. So my first question is, tell me about what type of research do you do? What type of data do you generally produce or collect?</p>
  <p><b>BSR09  </b>02:07</p>
  <p>Sure, yeah. I'm a postdoctoral researcher at [a university]. So I'm in [an engineering department]. So what we are what data we are working on is about like, the operation or the performance of infrastructure and environmental systems. So yeah, also, the scale of the data are at a city level. So we not only focus on the infrastructure, but also we look into the activities of the population because you know, the urban systems are always composed of humans and infrastructure. So the interactions between them are very important for us to understand the performance of the infrastructure or the the improvements that we needed to make to improve the infrastructure for future needs. Right. So yeah.</p>
  <p>Also, yeah, including social data, we, we usually collect data from the physical world, and also the virtual environment, like on a platform like Twitter or another social media platform. And for physical world data, we're usually looking to the human activities. So the like... damages in the road networks.</p>
  <p>So those are collected from like sensors or, like, human human powers are used to detect those kinds of damages on road.</p>
  <p><b>Sara Mannheimer  </b>03:41</p>
  <p>Yeah. Great. Okay, so I had asked you, I think I sent you an email a couple of few. It was a few weeks ago that we first interacted, but I asked you to identify a particular example of a research project you've</p>
  <p>done recently when you collected big social data, or reused social data that was collected by someone else. Do you have a project in mind?</p>
  <p><b>BSR09  </b>04:08</p>
  <p>Yes, we have a recently we have a project related to using social media data on Twitter to detect infrastructure disruptions. So the case is [a natural disaster in a U.S. city]. So yeah, we collect like about how many social media... maybe 20 mil ion or 10 to 20 mil ion Twitter data. And yeah, based on these data, we apply some natural language processing techniques to identify the content which are always supposed... which are related to the specific infrastructure in [the U.S. city], or [in city water sources]. We use... and also some specific roads or highways, toll roads, etc. and hospitals, airports.</p>
  <p>Yeah, a lot of infrastructure can be... the performance of this infrastructure can be assessed or sensed using this social media data.</p>
  <p><b>Sara Mannheimer  </b>05:21</p>
  <p>Okay. Um, tell me about your data collection method. Did you use an API?</p>
  <p><b>BSR09  </b>05:27</p>
  <p>Yes. We used a power track API provided by Twitter.</p>
  <p><b>Sara Mannheimer  </b>05:35</p>
  <p>Is that one that you purchased? That [your university] purchased?</p>
  <p><b>BSR09  </b>05:38</p>
  <p>Yeah, we have to purchase.</p>
  <p><b>Sara Mannheimer  </b>05:39</p>
  <p>Yeah. Okay.</p>
  <p><b>BSR09  </b>05:40</p>
  <p>They have the free API. The free streaming API. But, you know, the data is not complete if we use the streaming API. So we have to use the power track API.</p>
  <p><b>Sara Mannheimer  </b>05:53</p>
  <p>Yeah. Is the power track API complete? Or does it just provide more tweets? Are they talking...</p>
  <p><b>BSR09  </b>06:00</p>
  <p>It's complete.</p>
  <p><b>Sara Mannheimer  </b>06:01</p>
  <p>Okay, yeah. Nice. Great. And then was your project grant funded? Like? Did it require certain treatment of the data? Did you have a data management plan or any way that you are required to treat to handle the data?</p>
  <p><b>BSR09  </b>06:18</p>
  <p>Um, I'm not sure the funding sources for this data. But yeah, we definitely have like a, have a plan, or we have this actually, we have the server and we have the data storage account [specifics of data storage services redacted]. We use that to manage the data or share the data with teams. And also we use that platform to do the data computation, computational experiments.</p>
  <p> </p>
  <p><b>Sara Mannheimer  </b>06:48</p>
  <p>Okay. And then, last question about the data. Do you plan to share the data beyond your research team, the Twitter data?</p>
  <p><b>BSR09  </b>07:01</p>
  <p>Oh, the data is not allowed to be shared. Yeah, we have to keep data confidential.</p>
  <p><b>Sara Mannheimer  </b>07:08</p>
  <p>Okay. And is that an agreement that you made? I guess we'l  talk about this more throughout the interview. But is that an agreement you made with Twitter?</p>
  <p><b>BSR09  </b>07:15</p>
  <p>Yes.</p>
  <p><b>Sara Mannheimer  </b>07:16</p>
  <p>Okay. Through... because of the API? Because I have heard that you're allowed to publish just the tweet IDs. So just like the ID number?</p>
  <p><b>BSR09  </b>07:26</p>
  <p>Um, I think that since the data is... since we purchased this data, so we are not even allowed to share the ID.</p>
  <p><b>Sara Mannheimer  </b>07:35</p>
  <p>Okay. And do you plan, how long do you plan to store and retain the data? Do you know?</p>
  <p><b>BSR09  </b>07:48</p>
  <p>We can store the data for, for us, I think.</p>
  <p><b>Sara Mannheimer  </b>07:54</p>
  <p>Yeah, yeah. And do you think you wil  store it like, wil  it continue to be useful to you and your team?</p>
  <p><b>BSR09  </b>08:00</p>
  <p>I think we should probably hold on to this data for five years. And after five years, we may, because</p>
  <p>[after it has been] five years, I think no one would be interested, that we'l  definitely switch our research directions or topics to other disasters or other contexts.</p>
  <p><b>Sara Mannheimer  </b>08:28</p>
  <p>Yeah. Great. Okay, so I'l  start with my first of the six issues context. So I have a little quote to help sort of communicate what I mean by context. It's been suggested that "when data is col ected from social media platforms, just like when you collect data from people in traditional spaces, context is important.</p>
  <p>However, the context of a social media post could be absent or difficult to understand, because social media posts are by nature, short pieces of text, or images or videos that are taken from a larger context of a person's personal and public life. And then this out of context effect could be compounded when data are masked at this very large scale, mil ions of tweets." So can you tell me about a time during this research when you considered the issue of maintaining and understanding the context of the tweets that you collected? So like contextual information about the community where the data was collected or about the respondents or about the situation they were in, for instance.</p>
  <p><b>BSR09  </b>09:42</p>
  <p>First, I think when we start to collect the data, we are usually using several filters like keywords that related to [the natural disaster] or the the geotags in the tweets, which can help us to locate the tweets posted by the people, in [this U.S. city]. So, this helped us a lot to understand the use of tweets posted or related to the context that we are trying to study. Also, we use natural language processing approaches to, to filter out some tweets that does not make any sense, where there are some tweets only, like a post, punctuations or emojis, or some tweets do not have any, like contextual information, or like other keywords that can make sense to us. So, we remove those tweets, so we only focus on tweets that can deliver effectively information to us related to the disaster.</p>
  <p><b>Sara Mannheimer  </b>10:53</p>
  <p>Did you have any, like as a research team, as you were working through your research design, did you have any concerns about context about how these what the meaning of these tweets would be and how you sort of pulled them together to create a bigger picture of what people are thinking during [the disaster] that you can connect with your infrastructural data?</p>
  <p><b>BSR09  </b>11:25</p>
  <p>I think in our research, there wil  always be concern about the context. Because you know, even some contexts that use... you may think it's related to the project or related to other problems that we are trying to solve. But maybe it is posted for other cases or other contexts or other events. So it is... it wil always be a concern. But we use the filters to try to reduce the uncertainties from this kind of perspective.</p>
  <p><b>Sara Mannheimer  </b>12:08</p>
  <p>Did you consult with anyone... like what resources did you consult, or people, to come up with this strategy that you ended up using of like filtering in order to support context?</p>
  <p><b>BSR09  </b>12:22</p>
  <p>To support context, we have to look at the inverse effect the area of the events like [this natural disaster] or a more recent event, which is [another natural disaster], right. And also, from the results we wil  also compare this with our results generated from the Twitter data and also our model with information collected from news articles. Because usually news articles are confidential, or not confidential, are trustable. Right. So, you know, we use information from news articles and government reports to validate the information we gathered from Twitter.</p>
  <p><b>Sara Mannheimer  </b>13:06</p>
  <p>Oh, interesting. Okay, we're going to talk more about comparing different data sets too. So actually, let's just talk about that now. So during this research, did you compare or combine multiple datasets?</p>
  <p>Like when you used the news articles was that a data set that that you then computationally compared with the tweets?</p>
  <p><b>BSR09  </b>13:29</p>
  <p>Actually, we do not do the quantitative comparison. We just use news articles to validate the results we obtained from Twitter. For example, we use the Twitter data to map the timeline of events in a specific neighborhood, you see, [when the natural disaster is more or less impactful], when the power is out, and when the power is recovered. So we use the Twitter information to to to to map the timeline of the events in the neighborhood, then we correct the correct news article to find the EFCC information are consistent with the information posted in the news article.</p>
  <p><b>Sara Mannheimer  </b>14:13</p>
  <p>Oh interesting. Did you do any other type of combining of data in this project?</p>
  <p><b>BSR09  </b>14:20</p>
  <p>Um, I think we only use the Twitter data. So we do not do any combination.</p>
  <p><b>Sara Mannheimer  </b>14:27</p>
  <p>Have you ever done that in any project?</p>
  <p><b>BSR09  </b>14:34</p>
  <p>In the past project, if we want to combine data, we only combine data which are compensated to each other. For example, we have the Twitter data and also we have like a point of interest where the data is used, and then we can integrate them to look at where the tweets were posted around the which area and [what damages are occurring in that place]. Also, we may compare it with combined with social demographic data to see which area are populated, or are more concerned by the user on Twitter, but we did not turn by any other similar data sets together.</p>
  <p><b>Sara Mannheimer  </b>15:20</p>
  <p>Yeah, okay. Did you.... Okay, I see. Sounds good. Okay, let's move on to data quality, and trustworthiness. So can you tell me about a time during this research when you consider the issue of data quality? For example, missing data, or bots, or bias?</p>
  <p><b>BSR09  </b>15:46</p>
  <p>Yeah, we considered the bots on Twitter actually. So to say, there might be some bots, but we may not be able to identify them manually. So there is a tool available online, which is publicly free tool. And if we only want to test a few tweets, it should be free. And if we want to test mil ions of tweets, we have to pay for something. So us-, using this kind of tool, we can detect the accounts which might be the bots, and then we remove the tweets posted by these accounts.</p>
  <p><b>Sara Mannheimer  </b>16:32</p>
  <p>Okay. What other did any other data quality issues arise during the research?</p>
  <p><b>BSR09  </b>16:44</p>
  <p>Another issue might have been geotags on the tweets, you know, usually there's a, around like one or 2% of tweets are geotagged. So if we want to assess the area based on the... or assess the damages of the area based on geo tagged tweets, it wil  definitely have a lot of like quality issues, because usually people don't like to post the tweets with the geotags. So so to do to, to like overcome these issues, we developed a method to extract the locations from the content of the tweets, then we locate these tweets in the specific area, which is a mentioned as an address. So we can use this to locate the the tweets in the area, and they use that tweets to assess the damages in the area.</p>
  <p><b>Sara Mannheimer  </b>17:42</p>
  <p>Were there enough tweets? I mean, I guess you did collect mil ions, but were there enough that you were able to extract a location from enough of them?</p>
  <p><b>BSR09  </b>17:51</p>
  <p>I think, is stil  not sufficient. Yeah, but it's improved. A lot of which is only based on the geotag tweets.</p>
  <p><b>Sara Mannheimer  </b>18:03</p>
  <p>Yeah. Um, and did you... thinking about data quality. Like, did you consult with anyone on this? Did your team talk together about these issues? And yeah, tell me more about what the process was?</p>
  <p><b>BSR09  </b>18:21</p>
  <p>Sure. Yeah, we usually discuss with our own team members. And we also look at the literature, which has been published before. But actually, previous literature does not deal with these issues very well.</p>
  <p>So we have to come up with our own ideas, or approaches to, to somehow overcome this issue. Yeah.</p>
  <p><b>Sara Mannheimer  </b>18:49</p>
  <p>Yeah. And then, how did you like communicate these strategies that you had used? Did you write about them in your paper or otherwise sort of document these strategies that you use to support data quality?</p>
  <p><b>BSR09  </b>19:07</p>
  <p>Yeah, we always describe this approach clearly in our published papers.</p>
  <p><b>Sara Mannheimer  </b>19:14</p>
  <p>Yeah. Okay. Let's move to informed consent. So can you tell me about a time during this research, if any that you thought about informed consent of the participants, Twitter users?</p>
  <p><b>BSR09  </b>19:38</p>
  <p>Oh, could you repeat that, sorry.</p>
  <p><b>Sara Mannheimer  </b>19:40</p>
  <p>Yeah. Was there a time during this research where you thought about the idea of these users agreeing to be part of the research? consenting to be part of your research?</p>
  <p><b>BSR09  </b>19:52</p>
  <p>Because, you know, Twitter has, like a consent agreement with users. So as long as they sell the data to us, we can use that. So we didn't have any concerns about this.</p>
  <p><b>Sara Mannheimer  </b>20:07</p>
  <p>Yeah, it's interesting that you... that Twitter sold the data to you. And so it was, you were actually sort of making an agreement with Twitter to, to download the data. And then did you talk about that at all with Twitter? Or did your research team? Or was it just we have permission to give these tweets to you, and therefore...?</p>
  <p><b>BSR09  </b>20:32</p>
  <p>Twitter has agreement with the users. So they have this information on their website. And then when we purchase data, we are also assigned a sign a very strict agreement. They mentioned that this kind of agreement with our users and with us, so yeah, we talked about that.</p>
  <p><b>Sara Mannheimer  </b>20:55</p>
  <p>Yeah. Did you do you as a researcher, like, have any questions about that? Or did it seem like that made sense? And you thought that the participants would be okay with the research that you're doing?</p>
  <p><b>BSR09  </b>21:13</p>
  <p>Yeah, we actually, we thinking about that. And also, we submitted our IRB application to get approved.</p>
  <p>So yeah, we... any publications are based on the approval of IRB. So yeah, we definitely think, are thinking about these things, thinking about that, and we got approved. And yeah.</p>
  <p> <b>Sara Mannheimer  </b>21:39</p>
  <p>Okay, sounds good. How about privacy, and confidentiality. So this is the fifth out of six. Can you tell me about a time during this research, when you considered issues of privacy for you know, like protecting the data, or protecting the data during your research? or thinking about the people whose tweets you're pulling? Like, did you publish any tweets in the paper? any full text tweets or...? What issues arose as you thought about privacy of Twitter users?</p>
  <p><b>BSR09  </b>22:20</p>
  <p>Okay, I think there are two issues we always keep in mind. The first issue is user name, or user ID</p>
  <p>information. Also, also, although we analyze the tweets using like a social network analysis, but we never show any IDs in our publications. I think, as you see, it's very important to protect, like the privacy of the users. And another issue is, I think it's the, like content in the, in the picture. Sometimes people may have a face in the picture. And we, we don't want to show these in that, in our paper. So we usually remove any of the results or any of the, like, [any data] that contained a photo of any people. So this is what we... what actions we take to to protect privacy.</p>
  <p><b>Sara Mannheimer  </b>23:23</p>
  <p>Is that like a policy in your lab? Like, did you... Is that something that... just wondering, like how you came up with that? You know, thinking about how Twitter sold you the data, and said all these people had consented for their data to be used? So and then the IRB said you're exempt. How did you come up with these strategies? Why did you decide to use these strate-, these privacy protection strategies?</p>
  <p>And then how did you come up with what you would and wouldn't do in terms of protecting privacy?</p>
  <p><b>BSR09  </b>23:57</p>
  <p>Okay, thank you. And I think that this process was started because of the IRB issues mentioned in previous papers. So some other researchers mentioned that and also, when we do the training to be a researcher in the university, they also mentioned the kind of, like, rules we have to follow. So both university training and also the previous papers, tell us we should do that.</p>
  <p><b>Sara Mannheimer  </b>24:31</p>
  <p>Are those previous papers that you published or that you've read?</p>
  <p><b>BSR09  </b>24:35</p>
  <p>Other researchers.</p>
  <p><b>Sara Mannheimer  </b>24:36</p>
  <p>Yeah. And so did you discuss it within your research team? and say, "Okay, these are the ways that we're going to protect privacy," and did you end up publishing any parts of tweets in the paper? Sorry I'm asking a bunch of questions at the same time, did you consult with any about besides sort of reading different papers, and like the university training or whatever type of training you did, were there any other consultations that you did within your team or like with a librarian or anybody about, about privacy and how to support privacy with social media?</p>
  <p><b>BSR09  </b>25:20</p>
  <p>Um, definitely, we discussed it with my advisor, and my advisor may discuss with librarians or other staff to talk about this issue. But actually, I do not have too much experience talking to other people.</p>
  <p>Yeah.</p>
  <p> </p>
  <p><b>Sara Mannheimer  </b>25:35</p>
  <p>Yeah. And then did you? What did you end up publishing? Did you publish any quotes or examples of the tweets in your paper?</p>
  <p><b>BSR09  </b>25:47</p>
  <p>Yes, we have some we included some content content on the tweets, but we never show like a phone numbers or address or any other information. In the paper we just show. For example, this house is damaged. We just chose this sentence. Yeah.</p>
  <p><b>Sara Mannheimer  </b>26:08</p>
  <p>Okay, so just like snippets of the tweets? Yeah. Okay. Awesome. Al  right. Last question. Intellectual property. Can you tell me about a time during this research when you considered intellectual property concerns? So you talked a little bit about the Twitter Terms of Service? I feel that's one of their Twitter's ways of protecting their... the intellectual property of the company, I guess. Were there any other concerns that you had around intellectual property?</p>
  <p><b>BSR09  </b>26:52</p>
  <p>Wondering how can we define intellectual property?</p>
  <p><b>Sara Mannheimer  </b>26:56</p>
  <p>I guess it would be like, well, with your example, because you purchased the data, I feel the intellectual property concerns are actually pretty cut and dried. But thinking about, you know, like, people, when you say, when a person tweets something, that's if I make a tweet, that's my own intellectual property, you know, like what I said belongs to me. But then Twitter has a license to post that on their website.</p>
  <p>It's like they, because of the Terms of Service, I've given them the permission to post it on their website, and I've also given them permission to provide it to researchers. But one of the issues that I've seen as I've conducted this research is that Twitter users don't often fully read the Terms of Service and don't always completely understand what they're agreeing to. So I'm just wondering if....</p>
  <p><b>BSR09  </b>27:55</p>
  <p>Oh sorry.</p>
  <p><b>Sara Mannheimer  </b>33:19</p>
  <p>No worries. I was gonna write you an email, but I was like I think I'l  just wait. Well, we're nearly done anyway, so oh, I see, I got an email from you this week, I guess I was just sort of talking through all the potential intellectual property, I guess mostly with social media research, it's their social media Terms of Service, and then the intellectual property of the Twitter users. So it's okay to say that you didn't consider it too.</p>
  <p><b>BSR09  </b>33:52</p>
  <p>Um, I think the intellectual property might be the information they post related to their emotions or their reactions to the disasters. So we actually we have a paper to study the emotion of the users on social media to see how the emotion change can reflect the damages of the infrastructure systems.</p>
  <p><b>Sara Mannheimer  </b>34:23</p>
  <p>Oh, I'l  check that one out. Okay. Al  right. Great. Well, are there any other issues or challenges that arose during this research that I haven't asked you about? I think that's al  I have on my interview guide.</p>
  <p>I have a last question about looking for other people to interview but you're one of my last interviews so</p>
  <p>I don't need any more participants. Thank you so much. I really appreciate it. And good luck with it, the rest of your research and the rest of your postdoc.</p>
  <p><b>BSR09  </b>35:03</p>
  <p>Thank you. Thank you. Good luck research as well.</p>
  <p><b>Sara Mannheimer  </b>35:07</p>
  <p>Thank you so much. Talk to you later.</p>
  <p><b>BSR09  </b>35:10</p>
  <p>Bye</p>
  <p> </p>


name: BSR_10
filename: Mannheimer_BSR10_Tra.pdf
created: 2024-07-11 14:35:33.075626
text_direction: LEFT_TO_RIGHT
contents:
  <p> BSR10_transcript_deidentified</p>
  <h2><b>SUMMARY KEYWORDS </b></h2>
  <p>data, tweets, people, twitter, fake news, collected, research, topic modeling, question, project, social media, bots, issue, develop, intellectual property, natural disaster <b>SPEAKERS </b></p>
  <p>Sara Mannheimer, BSR10</p>
  <p><b>Sara Mannheimer  </b>00:00</p>
  <p>Okay. So yeah I'm Sara Mannheimer. And that's my background I'm... the research, the main research question is how can we data curators best handle qualitative and big social research so that we can support responsible, ethical epistemologically sound, and legal share data sharing practices. And so the interview today is structured around six key issues which identified through a lit review. We have context, data quality data comparability, informed consent, privacy and intellectual property. And so I'l ask one question for each of those. And then I have an intro question and a wrap up question as well.</p>
  <p>So it should take us about one hour or less. So my first question for you is, introduce yourself and tell me about the type of research that you tend to do, and what type of data you produce.</p>
  <p><b>BSR10  </b>01:05</p>
  <p>Sure, currently, I'm an [faculty member at a university]. And I [am in X department]. Okay. And I, I have a background in computer science. And while working on my PhD work, [there was a large-scale crisis in the U.S.]. So I was a graduate student, and my advisor at the time, worked on a project about archiving al  these [posts about the crisis], like social media, usually Twitter, and along with the associated webpages that people share through Twitter, Twitter feed. So I became the GRA for that project, which lasted for many years. And I was a guardian to this in a world of social data collection, and analysis to some extent, and developing all these archives, by usual y webpage archives, based on social media data, by working with the Internet Archive. So I've been in that field, like uh crisis informatics for a while. And since then, you know, even after I came to [my current position], I, I could work with other people from like, all different departments like environmental science, social, social science, sociology, and political science and computer science, to to do a [grant funded] project about this using social media data, in addition to traditional like census data to identify all this, like, community resilience, when all these natural disasters hit. So I've been in this field for a while using a kind of social media data.</p>
  <p><b>Sara Mannheimer  </b>03:45</p>
  <p>That's so interesting. Can you so I guess, did you I had asked you at the beginning or when I first emailed, you have some time ago, to think of a recent time when you collected social media data or reuse data or prepared it for sharing? Did do you have an example in mind and we don't have to stick to it the whole time. But it helps to have a specific example as we talk through these questions.</p>
  <p><b>BSR10  </b>04:12</p>
  <p>More... the most recent case, we'l  be collecting Twitter data about [a recent national disaster].</p>
  <p>[Additional details redacted]. And you know, whenever I experienced or see this type of natural disaster I'm interested in like, how people communicate and who are the major people in this like social network,</p>
  <p>you know, sharing and trying to spread information and all this stuff. So I started collecting these Twitter feeds from [a certain date]. And [for about a month], and yeah, and then I am using this data like collected using keyword [company name redacted], which is an electrical company [in the state where the national disaster occurred], and also [the national disaster] hashtag, I could collected a couple hundred thousand tweets. So I started working on that data set to look at social networks and some topic modeling and stuff like that. Yeah.</p>
  <p><b>Sara Mannheimer  </b>06:04</p>
  <p>Um, so when you were collecting the data, did you have a specific research question in mind? Or were you just since this is sort of your area? Do you just collect and then come up with the research question, as you go...?</p>
  <p><b>BSR10  </b>06:17</p>
  <p>A research question? Um, what kind of a basic standard research quest-, question for me, in this crisis informatics wil  be like, I usually think about "what, when, where, and who" aspects of this disaster. And so, you know, this disaster had a before phase, a during, and an after phase, based on how much this, you know, [natural disaster] affected certain areas. And I, I'm interested in like what are the major, like, significant central people in this social network, and what they talked about and what kind of information they spread out. And, and their location information, too. So I wil  I plan to use, like, ArcGIS to plot their locations and see how that looks. And when wil  be like, three different phases? So, yeah, I usual y look at those as a baseline.</p>
  <p><b>Sara Mannheimer  </b>07:36</p>
  <p>Nice. Have you found that there, you can often find geographical information, like location information on the tweets?</p>
  <p><b>BSR10  </b>07:45</p>
  <p>You know, it's really rare that people trans-, turn on that feature. So I'm trying to rely on their user profile data. Because some people, you know, add their location data, like I'm in [city, state], and stuff like that. It would be like an approximation, but you know, no choice.</p>
  <p><b>Sara Mannheimer  </b>08:13</p>
  <p>Yeah. Okay, let's move to our first theme here -- issue, which is context. So just a quick sort of overview of what I mean by context. When we collect data from social media platforms, just like when we col ect data from traditional spaces, the context is important. But the context of social media posts may be absent or difficult to understand for, you know, for example, that location information might not be there other contextual clues might not be available. And also, social media posts are short pieces of text or images or videos that are kind of taken from a larger context of personal or public life. And then when you're amassing data at a large scale, it can become even more complex. So can you tell me about a time if any, during your data collection for this project, when you considered the issue of maintaining or understanding the data's context?</p>
  <p><b>BSR10  </b>09:17</p>
  <p>Context? That's a good question. Um, if we look at the data, like each individual, social media like tweets in a macroscopic way, then it's hard to understand sometimes because there wil  be a lot of junk posts, advertisement and fake news and fake news a big problem nowadays, and some people and bots wil  use a popular, popular Twitter like hashtags, just to advertise their product to so completely non-relevant content could be collected in the social media field. So I'm trying to look at in a little more macroscopic way to, you know, to do this, I usually apply like topic modeling, to see what kind of topics</p>
  <p>comes up from this, the whole collection of this Twitter data, like, hundreds of thousands of Twitter data. And yeah, so also, I typical y to understand some kind of large chunk of Twitter, I often use, what is the word.... a word cloud. Yeah, to quickly make the word cloud, and then see what kind of the main concepts and keywords people talk about?</p>
  <p><b>Sara Mannheimer  </b>10:05</p>
  <p>Yeah. Yeah. Do you...</p>
  <p><b>BSR10  </b>10:32</p>
  <p>....I understand the context.</p>
  <p><b>Sara Mannheimer  </b>11:03</p>
  <p>Nice. Um, do you do any, like, filtering out of bots, or bad quality tweets? Do you have sort of automated methods that you can use to sort of clean the bad data out of there?</p>
  <p><b>BSR10  </b>11:19</p>
  <p>There wil  be all-, always issue and I don't know for now, I didn't do much on that. That's one issue too.</p>
  <p>If I use a filtering, then I may remove some useful data, too.</p>
  <p><b>Sara Mannheimer  </b>11:42</p>
  <p>Right. So you were just use-, with the topic modeling. I mean, obviously, the bots and the fake news accountants wouldn't be top-, using talking about the same topics as others on the hashtag. So maybe they kind of filtered themselves out automatically. Oh, yeah. In that way.</p>
  <p><b>BSR10  </b>12:01</p>
  <p>If if I just run a topic modeling for the unfiltered, unfiltered tweets, I wil  have, like a couple topics. Al , you know, composed of like how you URLs, https and all these, because that's part of the tweet. So I know how to remove like URLs. And, and for those I don't know, I try to look at these a social network to see whether there's, like really influential entity in the network. And whether it is more like human being or bots, or if something else some organization by doing a social network analysis.</p>
  <p><b>Sara Mannheimer  </b>12:58</p>
  <p>Oh, so do you do all these strategies at once, like, do you do you do the word cloud? You do the social network analysis? And then also the topic modeling?</p>
  <p><b>BSR10  </b>13:08</p>
  <p>Yeah, I try to do that.</p>
  <p><b>Sara Mannheimer  </b>13:10</p>
  <p>Okay, yeah.</p>
  <p><b>BSR10  </b>13:11</p>
  <p>Because, uh, you know, this natural disaster is like, has a multi-faceted data set. So I want to see a little better picture from different angles. People involved in this and their topics. And their like, locations.</p>
  <p>Yeah.</p>
  <p><b>Sara Mannheimer  </b>13:36</p>
  <p>Yeah. Okay. location and network. Awesome. One more question, did you when thinking about these, like ways that you've sort of worked out to better understand the context of the data? Did you consult</p>
  <p>with anyone or look to other research projects? Or.... I don't know, look to any policies or guidelines, what's been your strategy to develop these methods that you use to sort of better understand the context of the data?</p>
  <p><b>BSR10  </b>14:09</p>
  <p>I think, yeah, over time, since my graduate student years, I could I could work well, all my projects were multidisciplinary. So I had many chances to learn from sociologists and environmental scientists, geologists and people from many different fields. So over time, I kind of developed my current strategy and a set of tools to look at this social media data.</p>
  <p><b>Sara Mannheimer  </b>14:49</p>
  <p>Great. Okay. We talked a little bit about data quality with the bots but can you tell me about a time, if there were any, during this research project when you considered the issue of data quality? So it could be like missing data or bias or bots, or the quality of the method to answer your questions.</p>
  <p><b>BSR10  </b>15:16</p>
  <p>For Twitter data unless it's purchased it's, as you may know, you know, it's a really small kind of random, small percentage of kind of random sample, if we collect them using a Twitter API. So well, at this point, I kind of use all of them, even though, you know, there wil  be some posts by bots, I kind of include them as part of the data set and see whether but could be, you know, kind of influential central entity in the social network. And in most cases, if, if it's not become so popular in the network, then it'l be kind of, you know, not distinguishable from other central people central figure. And if they are identified as one of the central figures in the network, then I want to look at it in more closely and see how they do and stuff like that.</p>
  <p><b>Sara Mannheimer  </b>16:33</p>
  <p>Right. Like there might be a bot that's tweeting out like the most up to date [information relevant to the disaster] or something like that. And that does, that wouldn't, that would stil  be something you were interested in. Okay. Any other data quality issues that you've encountered during your work?</p>
  <p><b>BSR10  </b>16:50</p>
  <p>Twitter is full of junk, junk messages, too. And, well, recently, I specifically looked at the fake news.</p>
  <p>Yeah, this information and misinformation occurring in Twitter. So I had that as a separate study. And, yeah, based on these snopes.com, the list of fake news, during a certain disaster. And specifically collected a list of fake news that were spreading during this [specific disaster]. And then I look at those social media cascade stream of these tweets connected with each other specifically for that disaster.</p>
  <p>And yeah, so I look to</p>
  <p><b>Sara Mannheimer  </b>17:57</p>
  <p>Did you find-...</p>
  <p><b>BSR10  </b>17:58</p>
  <p>Oh, sorry.</p>
  <p><b>Sara Mannheimer  </b>17:59</p>
  <p>Did you find that those tweets influenced other tweets? As you were doing that research? The disinformation?</p>
  <p><b>BSR10  </b>18:09</p>
  <p>Uh, tweets influence other people?</p>
  <p><b>Sara Mannheimer  </b>18:14</p>
  <p>Yeah, like, did the tweets with disinformation or misinformation, get spread? And then did it influence? I guess, I don't know. What what were your results, was all I'm asking.</p>
  <p><b>BSR10  </b>18:24</p>
  <p>Oh, yeah. Results. Um, you... I could see many political political tweets. Yeah, from like, around the</p>
  <p>[year when the disaster occurred]. There was like, a lot of, like, stuff by, like, left and right, by conservatives and more democratic, Republican and all these things, so I could see some fake news generated by far right and far kind of far left also. And, you know, a lot, a lot of stuff about [details redacted]. These uh, activists saying that they were some people were blocking right process. Some recovery activities and stuff like that.</p>
  <p><b>Sara Mannheimer  </b>19:45</p>
  <p>Hmm. Interesting.</p>
  <p><b>BSR10  </b>19:48</p>
  <p>Yeah. And usually those tweets become really popular in the first one or two days. So if you make a graph it just skyrockets, like in the first couple of days. And then after two or three days later, I could see that people who are skeptical about the news may post this on Twitter, like, referencing a fact-checking website.... Oh, actually what you're talking about is fake news. And they were trying to correct those invasive spread of misinformation. So I thought that's an interesting kind of self regulatory activities behaviors in a social network.</p>
  <p><b>Sara Mannheimer  </b>20:43</p>
  <p>Yeah.</p>
  <p><b>BSR10  </b>20:44</p>
  <p>So yeah, that's one interesting thing from the fake news and social media.</p>
  <p><b>Sara Mannheimer  </b>20:49</p>
  <p>Wow. So interesting to be able to study this. Wel , I feel like this is somewhat related, or the next issue is data comparability. So in the [two examples you've discussed so far], did you compare or combine multiple data sets? So you said you used Snopes information during the [second] study? Can you tell me more about like, if you encountered how, what challenges you encountered when you're been trying to combine multiple datasets?</p>
  <p><b>BSR10  </b>21:27</p>
  <p>Combine multiple data sets...</p>
  <p><b>Sara Mannheimer  </b>21:29</p>
  <p>Or use them in, in conjunction like you would talked about using census data with the Twitter data and stuff like?</p>
  <p><b>BSR10  </b>21:38</p>
  <p>Well, combining well in social Twitter data, usually we use, you know, multiple keywords and multiple hashtags generated for a specific natural disaster. So for example, I wil  use a hashtag [hashtag redacted], either keyword [keywords redacted], some other keywords too. And then once we develop all</p>
  <p>these keywords, I the last thing, the next thing wil  be to kind of merge them together. Because people use multiple hashtags and keywords at the same time. So merging was the one issue.</p>
  <p><b>Sara Mannheimer  </b>22:25</p>
  <p>Was it difficult to merge because of the size of the data? Or what were the difficulties there?</p>
  <p><b>BSR10  </b>22:31</p>
  <p>Well, yeah, the size of the data, somehow we need to put it in the data database. And by using just probably spent, well, if we use a database, technically, it's not a too much of a big deal, because each tweet wil  have its own tweet ID. So we can, you know, color code it so that the database can have only unique tweet IDs. So but you know, how if you work with people, different institutions, for example, for either of these collections, I was trying to share them with [a different university], some of the people [at that university], because they, they could work with the Internet Archive, and I was trying to help them develop web page collections. So in the case here, it's big data, somehow I needed to put it somewhere like in the cloud or in my own server, so that people can download that. At least, you know, Twitter IDs. So yeah, I'd say size of the data matters sometimes.</p>
  <p><b>Sara Mannheimer  </b>23:59</p>
  <p>Yeah. How about when you're using like census data? Or Snopes data? In addition to the Twitter data?</p>
  <p>Do you encounter any challenges there?</p>
  <p><b>BSR10  </b>24:16</p>
  <p>I don't personally use a census data.</p>
  <p><b>Sara Mannheimer  </b>24:20</p>
  <p>Okay.</p>
  <p><b>BSR10  </b>24:21</p>
  <p>Yeah, but my yeah, my team members use the census data. Yeah. And for using snopes.com. For the</p>
  <p>[natural disaster], we have like mil ions of Twitter data collected. And then I wanted to find the specific tweet, that reference of specific fake news. So I have to search through al  these Twitter data in in my database. Then to do that snopes.com wil  have its own kind of title for that fake news event. Okay. But if you use that title as is, you wil  have a kind of... How would I say? you wil  only collect tweets that reference snopes.com exactly as the title says. So I had to develop some strategy to use some synonyms of some certain keywords of these titles of the fake news and stuff like that.</p>
  <p><b>Sara Mannheimer  </b>25:45</p>
  <p>Yeah, nice. So, um, was this, the fake news or the disinformation study you did with [the disaster]? That was on data... Because the article that I had found you through was this one where you're looking at police activity during [the disaster], police activity on Twitter? So did you use the same data set for those two? Is it like a big data set [about the disaster] that you've collected, and then that you're using for different purposes?</p>
  <p><b>BSR10  </b>26:12</p>
  <p>That data was smaller for that data, I collected it myself using a tool. And for the fake news data, we purchased the [disaster] data, with our funding, from Twitter. So that was a more more complete kind of data set. Okay, for the police study, I use the data set that I collected using the Twitter API.</p>
  <p><b>Sara Mannheimer  </b>26:46</p>
  <p>Nice, okay. Yeah. Cool. Al  right, let's move to informed consent. Can you tell me about a time if any, during this research when you considered informed consent, participants so to speak like the Twitter users?</p>
  <p><b>BSR10  </b>27:07</p>
  <p>Well, not specifical y for me, but a one of my team members, project team members, they use informed consent when they want to... when they were trying to do surveys for the people. And they collected the contact info, these people's name or contact information from the Twitter data set. So it started from collecting the Twitter data set. And they searched through some of interesting figures in Twitter, Twitter, including not only individuals, but also organizations, and like NGOs, [and other groups active in the area where the disaster occurred]. And then when they wanted to conduct surveys, they went through all these informed consent and stuff like that. Yeah. Yeah.</p>
  <p><b>Sara Mannheimer  </b>28:10</p>
  <p>And contacted them? Okay. Yeah. Tell. So did you go through IRB or any processes when you were doing this Twitter research? Or were you just thinking of it as existing data that didn't require human subjects treatment?</p>
  <p><b>BSR10  </b>28:25</p>
  <p>Yeah, Twitter data. I didn't go through IRB.</p>
  <p><b>Sara Mannheimer  </b>28:29</p>
  <p>Yeah. And did you? Like, did you feel that the participant or the Twitter users whose tweets you collected...</p>
  <p><b>BSR10  </b>28:42</p>
  <p>Yeah.</p>
  <p><b>Sara Mannheimer  </b>28:42</p>
  <p>Do you feel like they would expect to be used in or in, in research like this? Or what did you feel their expectations were as far as being part of research or not?</p>
  <p><b>BSR10  </b>28:57</p>
  <p>It's a kind of a debatable topic.</p>
  <p><b>Sara Mannheimer  </b>29:00</p>
  <p>And that's why I'm asking about it.</p>
  <p><b>BSR10  </b>29:04</p>
  <p>Well, if I go back to another study, we, with another, my colleague at [my university], we collected a tweet posted by the libraries in [disaster areas] and then wanted to look at how they communicated during a certain like [disasters]. And there were some just library patrons communicating with these libraries. So in the case, when I publish things, I try not to focus on these individuals.</p>
  <p><b>Sara Mannheimer  </b>29:49</p>
  <p>Yeah.</p>
  <p><b>BSR10  </b>29:50</p>
  <p> And try not to talk about what they said and stuff like that at all. And my focus wil  be more on like, you know, these more public entities, like institutions, federal entities, like public libraries, and FEMA and stuff like that. So even though I use Twitter, and maybe some of these, their tweets wil  be contributing to my topic modeling study. But I try not to talk about, like individual tweets, exposing their private information like that.</p>
  <p><b>Sara Mannheimer  </b>30:31</p>
  <p>Okay. So you don't do any, like quoting of tweets, in your articles?</p>
  <p><b>BSR10  </b>30:39</p>
  <p>Quoting?</p>
  <p><b>Sara Mannheimer  </b>30:42</p>
  <p>Like, would you say, you know, this person said, "[disaster is occurring], and my electricity is out? Or something like that, you know,</p>
  <p><b>BSR10  </b>30:49</p>
  <p>I try not to, at least don't provide the, you know, like their Twitter IDs, or something like that. I try not to, unless it's really...</p>
  <p><b>Sara Mannheimer  </b>31:01</p>
  <p>Tell me more about like that thought process? How did you come to that decision to sort of focus more on the institutions rather than individuals?</p>
  <p><b>BSR10  </b>31:11</p>
  <p>Yeah, it's a well, now, you know, Twitter provides the Twitter data for free for research purposes now.</p>
  <p>And so that means, you know, as long as we abide by their terms of service, and we can use the data set in a research and but well, I've attended the training for data curation provided by [a university] a while ago. So including those experiences, I learned about, you know, this, protecting privacy is, you know, really important matter in your kind of social social data set, or any, like medical data set to specifically. So by also talking, discussing with my colleagues, I could sort of develop that. Not only myself, but you know, it's kind of a team team effort. I uh, I think we, we should better do this way or that way.</p>
  <p><b>Sara Mannheimer  </b>32:32</p>
  <p>Yeah.</p>
  <p><b>BSR10  </b>32:33</p>
  <p>Over time.</p>
  <p><b>Sara Mannheimer  </b>32:34</p>
  <p>Yeah. Okay, that's great. That kind of led into my next question is about privacy. So oh, maybe we've covered this. But do you have any other thoughts about issues of privacy, like protecting the data during research? Or do you ever make your the Twitter datasets you have available to other researchers? Or have you ever thought about publishing these datasets?</p>
  <p><b>BSR10  </b>33:02</p>
  <p>For Twitter, we, in my project team, we share we purchased maybe total three data sets from Twitter about [three different natural disasters], and then since we talked about, we wil  share some of our data</p>
  <p>set. When we submitted our proposal to [a funding agency], uh, we share the tweet IDs but it's like 1%</p>
  <p>or 5% of tweet IDs and as a sample Twitter data so that people are interested in this could hydrate the full Twitter data.</p>
  <p><b>Sara Mannheimer  </b>33:54</p>
  <p>So you went... you got a grant from [a funder] to pay for the three data sets that you bought and so then was it because you weren't allowed by Twitter to publish the whole thing that you'd purchase the tweet IDs you just... they is that like from the terms of service they like only al ow you to publish a certain percentage of tweet IDs?</p>
  <p><b>BSR10  </b>34:18</p>
  <p>I don't think there's a restriction on that.</p>
  <p><b>Sara Mannheimer  </b>34:22</p>
  <p>Okay.</p>
  <p><b>BSR10  </b>34:23</p>
  <p>But that was our decision Yeah, if yeah, I believe we just to share the some sample Tweets and since one data set probably [certain disaster] was like 16 mil ion tweets, so 1% wil  be fairly many tweets. So yeah. Yeah. So if we just to share the, well, I'm more like, open sharing person. Yeah. But then after talking with with my other group members, we finally decided, let's share some small percentage of the actual data set that we use.</p>
  <p><b>Sara Mannheimer  </b>35:10</p>
  <p>And where did you share it? Did you share it in a repository? Or as a kind of...?</p>
  <p><b>BSR10  </b>35:16</p>
  <p>Oh, yeah. Through our project homepage. Yeah. I don't know if you're interested in that.</p>
  <p><b>Sara Mannheimer  </b>35:29</p>
  <p>Sure, if you have it handy yet, you could put it in the chat.</p>
  <p><b>BSR10  </b>35:32</p>
  <p>Yeah, let me show. Yeah, that's a specifical y data sets page. And we share the some of the web pages we collected by collaborating with [another university] and also, we have [tweets from other natural disasters]. Yeah, these are like samples.</p>
  <p><b>Sara Mannheimer  </b>36:02</p>
  <p>Nice. Perfect. Oh, that's cool. Al  right. Um, anything else you want to add about privacy? So like you said, your main approach to privacy was to try to talk about the tweets in the aggregate, but never speak of individual people or use individuals, quotes from individuals tweets and focus more on the institution. Any other like, thoughts that you heard about privacy as you've conducted these research projects?</p>
  <p><b>BSR10  </b>36:35</p>
  <p>And this is actually a big question to me too, because sometimes, for the fake news study, there were certain individuals who spread, who are spreading this false news on a big scale. Thousands of people were retweeting. So, at that time, I kind of said, oh, this person, I, I kind of talked about this person, saying, oh, this was a, this person is located in [country], and you know, based on that person's the</p>
  <p>public user profile data bio, that they typed in by themself. So I kind of, you know, I'm not on like, left or right. So I kind of described the situation, or this person, for some reason, posted this false news. And 1000s of people were retweeted in the beginning, but then I could see another group of people were sharing the debunking article from snopes.com after a couple of days later, and also some visited this person's Twitter page and left a message "Hey, you are spreading fake news." So I thought the whole event was interesting, like a self regulatory behaviors online. So in that time, I, like, I used this person's Twitter username, oh, [Name] blah, blah, and yeah. But if there is a I don't know, some other way that I don't have to do this. I wil  follow that.</p>
  <p><b>Sara Mannheimer  </b>38:31</p>
  <p>Yeah. Do you, like.... Are you aware of any like, have you looked at any guidelines, like when you decided, Okay, I'm going to publish this person's Twitter username. I'm like looking to hear more about people's thought processes, because there's just such a variety of ways that people think about this stuff. You notice there's no like real good... There's no IRB for social media data, you know?</p>
  <p><b>BSR10  </b>38:55</p>
  <p>Yeah.</p>
  <p><b>Sara Mannheimer  </b>38:55</p>
  <p>So how did you... how did you work through those ideas? Did you talk to people? Were there other similar studies that might have also used individuals ID Twitter usernames, or what was your? How did you come to that conclusion?</p>
  <p><b>BSR10  </b>39:11</p>
  <p>I usually I work with other people when I write a paper.</p>
  <p><b>Sara Mannheimer  </b>39:18</p>
  <p>Yeah.</p>
  <p><b>BSR10  </b>39:19</p>
  <p>So I talk to my collaborators first, which wil  be a better way to do this. And I also talked with my project team members about all these social media data. And always the dilemma... kind of a dilemma.</p>
  <p><b>Sara Mannheimer  </b>39:45</p>
  <p>Yeah, I am hoping through this project, we'l  be able to develop some sort of more standardized practices that curators and librarians can suggest. So this is really helpful. Thanks. Al  right, our last question about intellectual property. Can you tell me about a time, if any, during your recent research when you considered intellectual property concerns? So like, for example, looking at the social media Terms of Service or thinking about intellectual property for publications or news outlets or or people in the Twitter data set?</p>
  <p><b>BSR10  </b>40:30</p>
  <p>That I'm actually I haven't thought too much about this intel ectual property aspect of social media data.</p>
  <p><b>Sara Mannheimer  </b>40:40</p>
  <p>Yeah. Okay. That's a good, that's a good answer. But you have you did read the Twitter Terms of Service, you were saying, right, thinking about?</p>
  <p><b>BSR10  </b>40:55</p>
  <p>Yeah. And one thing is that they change often, they change often. So it's sometimes hard to keep up with the update. And when I talk to colleagues in other institutions, because I work with, you know, writing all these proposals, I work with people at [various universities]. So sometimes different people talk a little different stuff. So like, what, I don't know, some little bit outdated things. And, I don't know, for social-- sociologists may have a little better sense of this. And if I talk to engineers, and computer scientists, they were more on like a technical aspect of data collection and analysis. So less time wil  be given to looking at all these terms of service and stuff like that. Sometimes by working with the people who, who knows more about this intellectual property and privacy issues in detail, that helps as the project team as a whole, yeah.</p>
  <p><b>Sara Mannheimer  </b>42:29</p>
  <p>When you're putting together a project team, do you try and include, like a social scientist on it for that reason? Or do you mostly consult with them?</p>
  <p><b>BSR10  </b>42:38</p>
  <p>Um, project team? Well, I happen to join a project team for my prior [grant funded] project, through my</p>
  <p>[university]. And it's, I don't think I specifical y selected social scientists for that reason. And I bought, I used to work with social scientists who were interested in like crisis informatics and community resilience and those topics, so.</p>
  <p><b>Sara Mannheimer  </b>43:23</p>
  <p>Yeah.</p>
  <p><b>BSR10  </b>43:25</p>
  <p>But I don't specifical y pick people from sociology.</p>
  <p><b>Sara Mannheimer  </b>43:31</p>
  <p>Yeah. Interesting. Okay. Any other issues or chal enges that arose during your example, your research that I haven't asked you about?</p>
  <p><b>BSR10  </b>43:45</p>
  <p>Other issues? Well, stil , nowadays, some journals wil  ask you to submit the data set that you use the for the paper. So to some extent, I wil  kind of curate, like a Twitter data by myself, or only share Twitter IDs sometimes. But then sharing is a stil  a little issue, sharing it properly. Yeah, because, uh, I have my own server computer and I, I use a server or my laptop. So for a while, I'l  store this data in my server.</p>
  <p>But then, I don't know it's hard to do, like long time sharing. I can maybe use a, you know, good public data repository like Zenodo. But it it requires my time and effort which is uh, natural.</p>
  <p><b>Sara Mannheimer  </b>45:02</p>
  <p>Yeah.</p>
  <p><b>BSR10  </b>45:03</p>
  <p>Sometimes not easy to spend time on that.</p>
  <p><b>Sara Mannheimer  </b>45:07</p>
  <p>Yeah. Yeah, I'm experiencing that as I'm thinking about, oh, I want to publish these transcripts, I want to share my data and practice what I preach. But it is it's like you want to get your paper out? And if it's not a requirement, it can be tough. Yeah. Have you encountered any issues when you share your data on</p>
  <p>the website? Like, do people contact you? Or what's been your experience with those subsets of data that you've shared?</p>
  <p><b>BSR10  </b>45:37</p>
  <p>Not much issues actual y. There were people who downloaded our data set, but then not much issues.</p>
  <p><b>Sara Mannheimer  </b>45:50</p>
  <p>And how long do you tend to keep the your data? Do you keep it indefinitely? Like on your own personal servers?</p>
  <p><b>BSR10  </b>45:59</p>
  <p>Um, indefinitely? Yeah, at least three, four or five years?</p>
  <p><b>Sara Mannheimer  </b>46:08</p>
  <p>Yeah.</p>
  <p><b>BSR10  </b>46:10</p>
  <p>But at some point, I wil  just delete all the data I don't use and, yeah, I usually backup my data. But then after some years, I just clean out.</p>
  <p><b>Sara Mannheimer  </b>46:25</p>
  <p>Yeah, yeah. Okay. Well, I think that's it. You're my final interview.</p>
  <p><b>BSR10  </b>46:33</p>
  <p>Oh final!</p>
  <p><b>Sara Mannheimer  </b>46:35</p>
  <p>Yeah, thanks. I should be finishing by April. That's my plan. Analyze the data. Yeah. Yeah.</p>
  <p><b>BSR10  </b>46:47</p>
  <p>Good luck.</p>
  <p><b>Sara Mannheimer  </b>46:48</p>
  <p>Thank you so much. I really appreciate you taking the time to talk to me. It's been really interesting to hear your experiences.</p>
  <p> </p>


