Harvard: Experiential VR Learning in Education

Brad Scoggin: Well, today we are going to have a fun episode. We get to sit down with an old friend, Matt Cook. Matt Cook works with Harvard University Libraries as an in-house or as an in-house XR consultant. Matt, it’s great to see you. Thanks for joining us today.

Matt Cook: Yeah. Thanks guys. Thanks for having me.

Brad Scoggin: Yeah, absolutely. Well, I do want to hear about you. We met you when you were at the University of Oklahoma, and I want to hear about your kind of transition from there to Harvard, but maybe just a quick step back. You know, how did you get interested in XTAR to begin with?

Matt Cook: Yeah, great question. So my training, you know, academic background is actually philosophy, philosophy of mind, particularly. So what I studied in grad school is the way that tools can shape our thinking, and the way that we can extend our thinking with the tools around us, you know, the obvious, like, example is like a calculator, like you can’t do hard math in your head generally. So you use a calculator, but it still counts as, like, part of your cognitive process. That’s the way that philosophers break it down. Now,

Imagine the ad times a million, and you have VR. You have a complete virtual environment. Everything is under your control. You can put the things in that are relevant. You can take the things out that are distractions. So that’s sort of the theoretical background. And then at the same time, you have academic libraries, which are moving away from pure book collections. That’s not fun for undergrads anymore. So the idea here is now, and I’ve heard it described this way, knowledge in all forms.

So it’s not just books and totally unapproachable collections. You actually have the technology. You have tech loans. You have enhanced classrooms. You have all the things that you see in a modern university. So it’s kind of a cool opportunity to apply that mindset, which is the environment is a tool. And now we have this environment, an academic library, which is evolving. We can make it into a giant tool to support the thinking of undergrads, professors scholars. So that’s sort of where it all came together, I would say mid-2010s, 2015, 2016.

Brad Scoggin: Love that. Yeah, and we met when you were at the University of Oklahoma, and we did a few projects together. We were kind of together trying to push. It was early days, right? And it was all tethered headsets for the most part at that point, which I think at the time was just a massive barrier for VR and education. It was really dependent on labs and things. But yeah, tell us, I mean, so you’re at Harvard now, I mean, how has that been? What was that decision like, you know, making the jump from Oklahoma to Harvard?

Matt Cook: Yeah, sure. Believe it or not, I didn’t expect to be doing VR when I got to Harvard. I applied to a job that, you know, as someone that knew a lot about it, of course, I talked about it during the interview process, like that was my niche at OU, 3D and VR, but I didn’t see a whole lot of it happening here. Then immediately when I got here, I started getting requests to help out with VR implementation. So it was like, I couldn’t avoid it. Even if I wanted to, this is a super popular growth area across the world, you know,
universities. So I’ve been doing it a lot, and Harvard is doing it a lot. COVID sort of slowed things down a little bit.

But the thing about the Harvard versus the Oklahoma VR scene is we have just amazing sort of collection material. So you can think about rare objects that Harvard professors have spent hundreds of years going out and collecting. So everything from rare books to scientific instruments. We have 3D scans of birds in our Natural History Museum that we can put into virtual reality. So these are like kind of unique Harvard objects that we have access to that kind of give us the content that I didn’t have at OU. So that was a lot more geared toward out-of-the-box software and implementations.

Brad Scoggin: Yeah, I remember one of them, I think, I mean, you know, gosh, early again, VR in education. I think one of the examples that I used for a long time that heard from you when you were at the University of Oklahoma was, I think, I’ve always credited this to you, just the ability, you know, for like an anthropology class, typically they would get access to a skull or something maybe once a year. In VR, you get in there, and you can, you know, spin it around for hours if you want to, which is, which is really cool. And

So, but tell us, you’ve been at Harvard since 2019, right, for COVID, I know COVID has slowed things down, but over the last four years also, there’s been obviously a lot of new headsets at the market, a lot more content out there. I mean, how have things kind of evolved since you’ve been there with integrating VR into education?

Matt Cook: Yeah, totally. Whereas the tethered headset era in Oklahoma was all about like solving hardware problems, I think now it’s a content bottleneck that we’re facing. So the pain point is no longer the hardware, which is great. That’s like a problem that you can kind of force your way past with enough staff. There’s still issues, right? We all know just logging into a headset is not straightforward, depending on your vendor, but you can have people to help with that, and you can kind of get it so, and the ergonomics of the Tetherless headset are a little bit easier on the student as well. Transport, right, power, graphics, all that is like portable now. So we’re not worried about that. Now we’re worried about, okay, there’s too much interest, there’s too much demand. How do we build a custom application for everyone who walks in the door and wants to do VR? Because we’re hearing it in the news.

Right? Like Metaverse is in the news for better or worse. AR and VR are in the news for better or worse. As people walk in, they want to do something with this technology. Maybe they have no idea what. And, you know, building software is a big deal. It takes a lot of time, effort, and money. So we kind of have to pick and choose both who we work with and what we recommend to them. And I think that’s what’s changed. That’s not a problem. That’s that’s an opportunity. Right. But you just have to be familiar with like the environment, with the landscape of available tools.

Matt Cook: software. So I’m excited. I love the demand problem, and I like that we’re not fighting with software, but the new sort of challenge is content.

Will Stackable: I want to dig into that a little bit more. You know, you have students coming from all different walks of life and going into all different types of fields. There really is no one-size-fits-all solution. And I would say that’s true not just in education but also in industry. We talk to companies that are using it for five or six different types of applications. So for you in your context, how do you create an XR program that scales and uses these new technologies effectively with such a range of disciplines and backgrounds?

Matt Cook: Yeah, great question. So I’ve basically doubled down on a small number of tools, productivity tools essentially, that exist in XR but also have desktop clients. This makes it a lot easier to prep content, to provide an accessibility option for those who are not comfortable in headsets, and also to do cross-platform multiplayer and engagement. So I’m thinking specifically of tools like Gravity Sketch

rapid prototyping and design. I’m thinking of spatial.io or hubs or Verbella or Engage for online classrooms where there’s a social component. Or, in the case of data visualization, there’s Virtualytics, they’re out of Pasadena. That’s sort of interesting.

big data analysis platform that you can use, drag and drop your spreadsheet into VR. Super powerful stuff. And if you have those three in mind, you can kind of help anyone because you can put a 3D model into hubs, or you can put a big data set into Virtualytics, or you can create a new product in Gravity Sketch. So it’s like, okay, we have something you can do to at least try XR in your classroom or with your research. And then, if you have the interest and you want to raise some money, maybe then we can build something custom.

Will Stackable: That makes sense. So there’s that broad spectrum of all different types of tools. I almost want to link all those in the show notes so people can do some Googling and some searching. Can you zoom us into a single-use case at Harvard? Maybe, whether that’s something out of the ordinary, like your ancient literature courses, or even you could also talk about, you mentioned some of the chemistry lab use cases that you guys are experimenting with.

Matt Cook: Yeah, absolutely. So coming from the humanities, like philosophy background, I kind of have a foot in that world and can speak that language. So a lot of what I do professionally is sort of translating the benefits of technology to a non-technical audience, specifically like a faculty member. So those are the most interesting use cases currently. I’ll just give you a few examples here. So early on pre-COVID, we did a classroom curricular integration called Staging Shakespeare. We have a scale replica of the Globe Theater in Harvard Library collections. It’s under lock and key. It’s in the basement of our special collections. We were able to 3D scan and place that object at human scale in virtual reality for these undergraduate students to feel what it’s like to perform on the stage, at Shakespeare’s stage.

Brad Scoggin: Wow. That’s cool.

Matt Cook: Right now, we have access to over a thousand 3D scans of ancient Egypt and the Near East through the Harvard Museum of the Ancient Near East. And that’s another faculty partner where we’ll be building sort of virtual worlds using these objects to populate scenes. So you can do research on sarcophagi or Giza pyramid artifacts or different ancient Egyptians like tombs that were unearthed early on when people were doing research on behalf of Harvard.

A couple of other examples, we did a Dante visualization. So you’ve heard of the Divine Comedy. It’s like heaven, hell, purgatory. We did a like essentially a reproduction of the engravings, the art, the original art that accompanied that work, you know, way back in the day. This was a very artistic stylized walkthrough for students to see sort of what the text was trying to communicate. So an atmospheric walkthrough of this Dante novel.

home, I guess you would call it from back in the day. And then, just today, I was on a call with a faculty member. The cathedral at Cluny or the monastery at Cluny, which is in France, was destroyed or fell to ruin more than 100 years ago. It was unearthed and recovered by another part of the faculty member, again, back in the day. He was able to make casts of these ornate architectural components

So we’re able to take those casts and create a scene for architects and students to study what it was like in medieval France, and these like in these monasteries. So these are sort of rare objects that are spread around the university that we’re using to build scenes or concepts that faculty are teaching that are difficult to communicate. Otherwise, we don’t have to travel anywhere. We don’t have to sort of create anything custom in these cases, but we’re able to really quickly and easily help.

Will Stackable: Wow, so multiple, yeah, so I wanna dig in on all of those, but I guess I have a broader question, which is we oftentimes talk about how VR is great for anything that’s difficult, dangerous, or expensive to simulate in real life. Where are you seeing, where is the advantage of using VR or AR in an educational setting, and what makes it better than traditional learning? Can you think what comes to mind?

Matt Cook: Yeah, totally. Absolutely. I get really deep into the literature where it’s like human-computer interaction. So like, for me, the lowest common denominator is when the scene becomes either too complex to view on a normal screen or on a textbook page, so there’s like too much detail, or you need to see something from multiple angles or perspectives. So if you have something like an architectural motif, so it’s like a column top, they call it a capital, right? Angles. This is not a thing that you can see on a flat surface. You need to be able to navigate around it to view it from multiple perspectives to understand what the people at the time were doing, what the culture was like, etc. These are perfect candidates for 3D scanning and visualization in virtual reality. Now that doesn’t speak to a whole range of sociological and cultural simulations like you discussed with previous guests, so my focus has always been sort of that have some place in the world that you’re sort of reproducing rather than like narrative content or simulation or training scenario. So there’s all these different approaches right to XR. If you’re studying some objects and it’s you’re looking at it in a textbook, then the chances are you could be doing it better in XR, and that’s sort of been my approach.

Will Stackable: I was gonna ask this later, but I feel like I have to ask now. As you’re talking about 3D objects and being able to really use them in a practical way in a virtual environment, I keep thinking about just image quality and being able to actually have a headset that matches the quality of the scan. What are your thoughts on the new Apple headset? And I know it’s really expensive, but I’m just thinking even with a Quest 2, it’s great, but are you really able to get the fidelity you need?

to be able to observe and zoom in closely and look at the textures and everything else. Or what’s kind of been the sweet spot for you in terms of quality versus price, if that makes sense?

Matt Cook: Yeah, great question. It looks, from all looks, you know, all the media, it seems like we’re getting pretty close to like human eye resolution with these headsets. Now, that’s never been the deal breaker for me. A lot of this is like practicality, the price being a big one.

Will Stackable: That’s crazy.

Matt Cook: That being said, I think we Will Stackable get to the point relatively soon where we Will Stackable reach the scanning level of resolution, which right now is higher, like you say, than the headset visualization.

But we don’t even need that much scanning to see patterns in relationships or that much detail to see patterns in relationships necessarily. So I’ve not encountered many scenarios where a person who didn’t see the level of detail that they wanted couldn’t move closer in a virtual scene or zoom in to achieve that level of detail even if the headset resolution was too low. So that hasn’t been the deal breaker, I think cost versus resolution is probably going to be a deal breaker. What I’ve been hearing is, and then interfacing. So what I’ve been hearing a lot is that eye tracking is the big deal with Apple Glass. So like, you don’t need a controller. Like when I work with a faculty member I mentioned, my goal is to translate the benefits to someone who may or may not understand the technology. So you put a game controller in a faculty member’s hand.

Matt Cook: It’s not good. It doesn’t usually end well, right? They, these are not everyone is a video game person. Not everyone understands why you might need a controller or how to use it without training. If you can just look at a menu item, eyeball it, right? And do a hand gesture and interact with a scene. That is a huge deal, perhaps even more important than scene resolution.

Brad Scoggin: Very interesting. So you’ve been in education, doing VR in education for a long time. We always ask this for our enterprise interviews, but seeing VR evolve, I know we talked about hardware and content. What are some of the other barriers or maybe the biggest barriers you’ve seen of trying to bring VR into education at the university level?

Matt Cook: Just moving, like practically speaking, moving headsets around between classrooms, keeping them charged, right? All the stuff that Arbor, I think, helps with, and I’ve noticed their kind of built-in functionality for some of these current headset management tools. So that’s one aspect of it. The other thing is sort of…

We can go down the content road again, but just sort of communicating the benefits in a way that someone can understand that without, I tend to go a little bit into the weeds with some of the jargon. So that can be problematic. Yeah, so in addition to just finding good content for someone that might be interested, not scaring them off with all the…

all the jargon and getting the headset to their class on time and making sure all the headsets are charged and logged in and they have Wi-Fi and the content is correctly preloaded and someone is not going to like, you know, get motion sick in the first five minutes and not have a backup plan. So these are all just like little things that kind of, you know, you notice over the years.

Brad Scoggin: What about the people side? You said when you went to Harvard, you weren’t planning to do VR. I mean, what’s it like selling it internally? Or is there a lot of selling leadership? I mean, what’s that process been like?

Matt Cook: I would say this, yeah, leadership.

I would say is less familiar with the technology. Colleagues, like peers, they get it. And those are the sort of people that brought me in. I’m so grateful to have them show an interest in the thing that I’m very passionate about and coming here and seeing that people are interested and we’re struggling with the same issues, so we can all sort of struggle together. So I do think there’s like a cohort of people that are doing it, doing it well, ready to go and ready to share and talk.

talk about it, and that’s growing. It’s particularly growing among, like, grad students so you’ll have like an instructor or grad student assistant that is like teaching a class, and they’re like talking to their lead professor, oh we should bring in this new technology from the library, and we’re gonna build a virtual exhibit for art history class in a gallery, in a virtual online gallery space and the students can place their own art and walk there as a group and discuss that art in real time so that’s like coming because they’re not going to go out of their way necessarily to do something new if they’ve been teaching the same class the same way for years, right?

Brad Scoggin: That’s cool.

Brad Scoggin (18:25.154)
Yeah. I think one thing we saw, well, yeah, one thing we saw back even when we worked together previously, Matt was, and we still hear this from those implementing VR education, just the challenge of integrating it within the curriculum, right? And like, you know, it’s augmenting, well, how do you fit it in and how does it, and I think you talked about earlier where we were going to talk about the organic chemistry example. And I think that’s a really, really good example. I’d love to hear maybe share a little bit about that process with you guys. And because I’ve got to think something like that too, for the faculty to see, wow, for the faculty to see, wow, there’s a real impact here.

Matt Cook: Totally, you’re right. In that case, we start with the syllabus. We work backward from the syllabus and try to force a square peg in a round hole. So you see there’s a week during the semester. Hey, wait a second. The students need to have a spatial comprehension of some molecule or set of molecules. And we know from research that this is a hard thing for people to visualize mentally. People really struggle to visualize the layout of atoms in space. It’s a complex object. So we’re going to fit in here.

Brad Scoggin: Yeah.

Matt Cook: And we’re going to just really narrowly intervene over the course of a week. We’re not going to try to rewrite the entire syllabus, and we’re going to throw at it every trick in the book. So we’re going to make sure that we, the faculty members, should not have to think about anything related to headset hardware or content preloading or any sort of delay, right? We want to just make it totally seamless out of the box. They shouldn’t see all the problems that we see and deal with.

It works, and we documented that it works by giving students surveys before and after class and asking them how they felt about their knowledge of what was being taught. It’s documented, you can read it yourself, the feedback was fantastic, the numbers are there, students are better off when they’re able to look at data in a way that makes sense, it’s natural, it looks like the thing in the world, blown up a million times and placed right in front of you.

Brad Scoggin: That’s awesome.

Will Stackable: And your team used Nanome, right? That was the…

Matt Cook: Totally, Nanom is cutting-edge stuff, and it’s used in industry, and it is being used widely. I’m really interested and excited to see what those guys are doing.

Will Stackable: Yeah, they’re a fantastic team, and whether you’re, you know, K through 12 universities or even it’s being used for pharmaceutical drug discovery, definitely worth checking out them. We’ll, we’ll link them in the show notes. I kind of want to, we’re getting, as we move to wrap here, um, I have a few things in mind, few areas we could go, but one of them, I want you to use the term digital superpowers before, and I know you went into this a little bit, but I want to give you an opportunity. What does that mean when you throw out that XR? It gives you a digital superpower. What does that mean to you personally, and then what does that mean for education more broadly?

Matt Cook: Yeah, that’s, I’m super excited about that concept because historically, people have thought that the best possible XR experience is one that recreated some physical experience perfectly. But when you start looking at like the sorts of objects that we’re talking about, you’re not allowed to do some things with them physically. You’re not allowed to stick your head inside a car engine, right? You’re not allowed to reach into a human body and see the way that blood vessels are distributed. You’re not allowed to go inside the Great Pyramids of Giza. Even if you traveled there, you wouldn’t be able to go inside. So why not give people powers that are not physically possible? Why not let them fly? Why not like some sort of haptic feedback?

These are all things that are happening in labs, like university labs, and being studied that are coming down the pipe that I’m very interested in. And it might be a good segue to just jump straight into the other sort of elephant-in-the-room technology, which is AI. I mean, this is linked intimately to VR and XR. So there are undergraduate teams right now at Harvard. Very interesting work. Voice to unity. You can speak new games into existence on the HoloLens. So that’s just what the undergrads are hacking together right now in one of our labs. But the way it’s been described is as a quote, post-symbolic communication. So text to 3D is a thing you can do now. You can write a line of text that describes an object. I want to see a pink elephant in a purple room playing pool. You can create a 3D scene with words. Now imagine if in the future you’re walking through. Millions of books in our collection, just walking down the row of, you know, stacks, they call them. And you have Apple Glasses on or something like it, Apple Glasses 4.0. You look at a binding on the shelf, and then AI takes the content of that book and creates a virtual world on the fly for you as a student to walk through what the author was talking about in that book. Now we have 20 million books at Harvard Library.

If you had AI plugged into the catalog in that way, you would have 20 million virtual reality applications. That’s the future of this technology in an academic library.

Will Stackable: Man, there’s a whole world there we could go into. We should do a whole other episode just on that. It reminds me of, there’s a great book by Neil Stevenson, Diamond Age. It really just talks about the potential future of education when you have advanced AI that you can have a conversation with that also understands and has context for the real world and how you could have the best teacher in the world. You could have Socrates, essentially. And you could make sense of this incredible amount of data that’s being generated past and present.

Does that, does that get you excited? I mean, this is a whole nother combo, but when you think about that intersection of AI and XR. I mean, some people call that the singularity or whatever. Um, there’s does that get you excited or does that terrify you? Or a little bit of both.

Matt Cook: It’s both, honestly, both. I think that they’re more closely related than people realize. Some people feel super, people in XR, I have noticed, can feel threatened by AI, but I think it’s the wrong way to look at it. AI is really, really great at automation. AI is the pinnacle of human automation and XR is the pinnacle of human interfacing. So you can get data to a certain point. For you can get a big data set to a certain point, you need human eyes to comprehend, interpret, to understand. So I love AI for the fact that you can narrow down a data set or sort of organize it, perhaps. But I don’t think it’s going to replace the experience of engaging with that data as a human being, as a student, or a researcher, or a faculty member. So you need both. Like, they may converge, like you say, at some point when the AI is in the visual cortex, let’s say.

Thank goodness we’re a little ways away from that, I hope. But right now, you still need to interface as a human being. And the best interface is one that is XR-related. And if there’s data on screen, it’s not good enough. If the data is refined by AI, it’s not good enough. You need some interface mechanism, and that’s gonna be the highest bandwidth one.

Yeah, I like what you said a few minutes ago about the digital superpowers. And I think that’s important. I think that’s important for people who are deploying VR and XR to be thinking about because it is so easy to kind of slip into, let’s just recreate what we have now, right? And I think you go back to when digital education kind of first became a thing back whenever, you know, it’s like, well, let’s make a.

PDF out of a textbook, that’s not helpful, right? I mean, it is helpful, but there’s so much more potential if you can start to think about what’s possible. So I love that. I think it’s important that people think that way. Maybe as we do wrap here, you’ve been doing this for what, seven, eight years. I mean, just some advice for someone’s first time bringing VR into their educational institution. What would you say?

Matt Cook: Yeah, so focus on, like we discussed, the syllabus and the objective study. So the objective study is like the thing that the students are learning about and then work backwards from that. And depending on what type of objective study it is, like if it’s a thing in the world, you’re in luck, you can either scan it or model it and put it in NextR pretty easily. If it’s a narrative or a training scenario, you have to think about production value, and that could mean fundraising. And then the sort of next generation that I’m interested in, I alluded to this before the call, and it’s AI-related, is generative.

XR, right? So this is where the humanists are benefited. There’s not a great way to get an English class, you know, we read books in English class, let’s say, let’s just generalize, or history class, you’re studying history. It’s like, all right, how do you go from just a body of text to a visualization that’s relevant? Well, maybe you could do it automatically if you have generative AI to 3D technology, and then all this huge number of academic departments that were not served by are now they’re open to it. So it does it can work for the humanities for this tech-centric discipline. STEM is easy I don’t want to say easy but you know it’s easy conceptually you still have to do it, and then you have these like the middle of the tier like productions right where you got to be a little bit careful, so that’s how I would think about it break it down into the object type think about what your production you know costs might be and then work closely with the faculty member.

Brad Scoggin: Mm-hmm.

Brad Scoggin: Yeah, fitting it into their existing plan. I mean, it’s interesting, it’s kind of, you know, the people component of XR adoption is so big. And I think, I mean, the advice you just gave is very specific to education, but we’ve heard similar advice for a pilot in enterprise. You know, it’s a similar type of thing, right? Like you’ve got, you need an aha moment from someone in leadership. And the way you get that is you serve their direct goals in a way that doesn’t distract from what they’re trying to accomplish. I mean, it’s, yeah, it’s a…

Matt Cook: Totally. Yeah.

Brad Scoggin: Well, this has been great, Matt. I really appreciate you taking the time. Like I said, it’s good to see you again. Where can people find you? LinkedIn, Twitter.

Matt Cook: Yeah, I’m on LinkedIn. I also have a program site, like a Harvard site, for what I do and what we’re working on here. I’ll send you guys both of those.

Brad Scoggin: Okay, cool. We can link them. So yeah, appreciate it, Matt. Good chatting with you.

Matt Cook: Excellent. Yeah, thanks.