Meta-Analysis

Join the Academic Skills Team for The Skills Pod. In this episode, Maths and Statistics Advisers, Matt and Shirley, are joined by Ioannis Kanakis, Associate Professor in Chester Medical School. They discuss what is meta-analysis used for, what kind of data is needed to perform it, and offer tips on how to overcome some common issues that students have. 

Follow us! @AcadskillsUOC

Don't forget to like, comment and follow the podcast!

Transcript

Welcome And Guest Introduction

Shirley 0:04

Hello and welcome to another episode of the Skills Pod. I'm Shirley John, Senior Academic Skills Advisor for Maths and Statistics.

Matt 0:13

And I'm Matt Roscoe, and I am a academic skills advisor also in maths and statistics.

Shirley 0:19

Today we're going to discuss meta-analysis, and to do that, we're joined by a special guest from the University of Chester Medical School.

Ioannis 0:26

Hello everyone. I am Dr. Ioannis Kanakis. I am associate professor in clinical biochemistry at the Chester Medical School, and I also uh lead the masters in diabetes.

Shirley 0:37

So

What Meta-Analysis Actually Means

Shirley 0:38

I guess a good place to start is uh what is meta-analysis?

Ioannis 0:43

Well, in in uh statistical uh uh using statistical terms, uh uh meta-analysis, which is uh uh reported in uh the vast majority of the systematic reviews, is a weighted average of effect sizes from individual studies that uh are being included in these systematic reviews. Uh and uh where the larger, or if you prefer, more precise studies get more weight.

Matt 1:14

Yeah, so so ultimately what what we're trying to do is take a bunch of studies that have been performed potentially all across the world and combine them together to find out what is the true state of knowledge as it exists now when we combine everything together into one result, is the way that I generally would think about that.

Shirley 1:32

Yeah, so I think the advantage statistically, isn't it, is that you are taking lots of um, well, some of them might be quite large, but potentially smaller studies and basically pooling them together, so you've got the advantage of a much larger um sample size. So yeah, metroanalysis allows you to get a bigger picture on what's going on, and as Johannes has said, it's um often part of a systematic review. But I think one of the things that sometimes confuses students I know is that I've had students say to me, Oh, I'm not doing a systematic review, I'm doing a metro analysis. And I think it's um worth making the point, isn't it, that metro-analysis, if you like, is a is part of a systematic review. It's not always possible if you don't have the right data, and we'll probably come on to that in a minute, but but actually it it adds to your systematic review with that statistics.

Matt 2:25

A nice quote that I saw somewhere, although unfortunately I've not been able to track down exactly where I saw it, is that meta-analysis has saved the lives of more people than you will ever meet. So to give it some sort of real-world context, it is actually about showing that treatments are effective and actually helping to treat illness. So it's always worth keeping in mind what is the end goal here. It's keeping people healthy as much as possible.

Shirley 2:48

Yeah, and I think um it also potentially showing what treatments don't work as well could be equally um important or which ones are not as effective sometimes. Um I know students are comparing two different treatments rather than perhaps treatment against control.

Why It Matters In Healthcare

Shirley 3:04

So uh I guess that sort of brings us on nicely to the kind of data that you might need, doesn't it? Really?

Matt 3:10

Yeah. So so typically uh we've been referring to uh two kinds of treatment, potentially. Certainly uh some sort of treatment and some sort of comparator. Generally, in least in our team, we would always say aim to have something with two arms at least in um so that you're looking at uh two different treatments. Um it is possible to do single treatment uh metro-analysis, but you are quite significantly restricted in what you can do. So if you only have a single intervention in a study, so to go on for this, what we would need is we would need to find studies that use similar interventions and then measure the effectiveness of those interventions in similar ways.

Ioannis 3:58

Uh well, there are there are two kinds of uh of approaches, the the two arms and the one uh one arm. And the the most uh um important uh aspect of the two arms is the randomized uh control trials, uh where we we do have uh an intervention and a comparator. Uh while the in one arm studies are mostly used for prevalence or rare events, but uh of course uh uh it requires uh careful uh handling.

Matt 4:33

Interesting that

Outcomes And Data Types To Use

Matt 4:34

you touched on the on the prevalence there. So that is where either an event happens to a participant in study or it does not happen to a participant in study. And so this is quite a common kind of data, which you can also see in two arm studies, sometimes that we call dichotomous data. So the die part meaning two. So there are only two possibilities: either an event happens or it does not happen. And so you'll see this quite a lot, especially with safety outcomes of a treatment. You know, so how often does an adverse effect happen rather than necessarily how well it performs in terms of uh treating the patients. So perhaps we could talk a little bit about the different types of data that we with that we see.

Shirley 5:18

Yeah, so we've mentioned dichotomous, and I think that that does come up quite a lot, doesn't it, with safety outcomes where either this particular adverse reaction happened or it didn't. But I think it also comes up in those sort of studies where you've you've got that sort of outcome that is based on perhaps a survival to a certain point, how many patients survived to a certain point, um, and how many patients did you have in the study on the particular treatment to start with. And so we would typically see the sort of statistics that you might see in the paper for that might be things like odds ratios or risk ratios. But the other type is where you're actually measuring something, and I'm not a medical person, so I think Joannis is probably better placed to say the sorts of things you can measure, but I've seen things like hemoglobin levels, for example, you might be measuring, or some sort of marker that helps you to understand whether the treatment is working. And so those would be continuous options where you might have things like means and standard deviations or some sort of measurement, or you might have means and confidence intervals um for each treatment um in your study.

Ioannis 6:28

So, yes, uh the the uh continuous uh uh data that we're using in systematic reviews basically are the uh means and standard deviations per group, and they are uh being analyzed using uh mean differences, or if uh uh sometimes sometimes the scales differ, so we we are using standardized uh mean differences, and in medical science, in most of the papers, uh uh uh these measurements uh concern uh biomarkers, for example, in terms of uh concentrations, or uh we can see sometimes data from imaging techniques, like for example, DEXA, where we uh assess the uh bone density of a patient in orthopedics. And uh yes, so uh uh these are the types of uh continuous data that we're using in systematic reviews.

Matt 7:25

Yeah, so so you you touched upon something that I was potentially going to mention as well, which was that sometimes you'll you'll see things measured in perhaps a slightly different way, but fundamentally it's still measuring the same quantity. So the example I wrote in my notes earlier was uh blood pressure. You'll sometimes see this in millimeters of mercury or some unit derived from Pascals. They're both ultimately measuring blood pressure, but the units are different. And so this is okay. We can use standardized mean difference to handle this if we don't want to do a conversion.

Shirley 7:55

Yeah, and I think this is where one of the kind of um issues can arise for students, isn't it, where perhaps different studies have used different ways to measure similar things and um potentially sometimes actually there might be work needed to do to put all the measurements from different studies into a form that can actually be used a meta-analysis to bring it together. Because I think that's worth emphasising, isn't it, that the meta-analysis has to have the same information from each study for the results to actually make some sense. So that sometimes can be a challenge, I think, um, and perhaps something that makes uh meta-analysis for systematic review um a bit more tricky for students to navigate where perhaps their papers and their studies they've included are doing lots of different things.

Ioannis 8:48

This is very important uh because, for example, uh I have seen many systematic reviews that uh have used studies that uh assess pain. So to assess pain, uh we use different questionnaires with different scales, etc. etc. So a student needs to pay attention to this and uh use uh at least similar uh uh questionnaires.

Matt 9:11

Yeah, but but also the the different ways of measuring the same kind of outcome. So sometimes we might see in oncology, we might see, for example, average survival time in some papers, whereas other papers might just count the number of participants who survived six months or a year or two years. Although they are measuring survival, they're not measuring the same thing. So this is where we have to be clear about are we dealing with dichotomous data or continuous. So in the example I just gave, the average time of survival is continuous, but the number of people who survived a certain amount of time is dichotomous. And these things are not really possible to mix, which is unfortunate. But depending on how many papers that you have, you may be able to do a better analysis with both of these things. Or sometimes what I'll do with students is I'll encourage them to look at supplementary material for papers to see if they could find any additional data that was not in the main paper that they can use to compare with the others. Because that's the key is getting it papers that measure the same thing. And you need a few of them, but not that many. So

How Many Studies Are Enough

Matt 10:21

this is a question that we get a lot, which Ioannis is probably better to answer here, which is how many papers should you have?

Ioannis 10:28

Well, there is no definite uh number of uh of papers, and of course, this depends on the research question and the discipline. Uh uh, for example, uh we have we can have a lot of paper of papers in oncology, uh in uh cancer, different types of cancers, but uh uh we can have uh a few uh RCTs in uh rare diseases, for example. So there is no uh definite uh number of how many papers should we uh include uh in a systematic review. Of course, the the higher the number of the patients uh uh it is uh it will give us a more safe conclusion. Uh but uh uh we need to pay attention to this because it can also add some uh kind of bias. So uh we need to have a a balance between uh those two. I would say in general uh that uh for a dissertation, for example, uh a good number of uh uh of papers would be between uh 8 and 15. Uh uh and uh to have uh a publishable uh result, uh an excellent dissertation that uh deserves to be um uh to be uh published, uh you can f you can have even more up to 20, 23, uh, or even 30 uh uh studies.

Shirley 12:09

I think sometimes students they worry that they've only got three or four papers, and I think that's partly down to, isn't it, the inclusion criteria? Is it does it need to be wider or um does it need narrowing if it if they've got too many? But um also I think there's a pragmatic decision students have to make, isn't there, about perhaps which outcomes to choose, and perhaps choosing the outcomes that they can see in as many papers as possible. But also I think realizing that actually maybe you can, if you've got a certain number of included studies, you don't need to have each every single one have the same set of outcomes. You can do the meta-analysis part with um, you know, a proportion, a subgroup of your um studies included, because you can still use other papers that perhaps don't have that particular outcome in a more narrative way to give a that bigger picture, even if it it can't be included than the meta-analysis. Um so I think it sort of brings us perhaps down to the nitty-gritty, isn't it? And particularly for a dissertation, you want to be pragmatic and choose outcomes that you know you can do a meta-analysis with that actually help you to answer your research question. Whereas perhaps the wider field of published literature is looking for the widest possible application.

Fixed Versus Random Effects Models

Shirley 13:30

And I think one of the other things we haven't mentioned in sort of the specifics of meta-analysis is that sort of thing about what type of model to use when you're bringing that together. And I think that probably brings us on to the software as well. But actually there's a choice, isn't there, between fixed effect models and random effect models. So fixed effect models are where you're kind of assuming there's a fixed effect that each study is trying to measure, whereas the random effect models allow you to include an effect for there being a I guess a random effect between studies. Um, I know we would always sort of um advise students to use random effects models. I don't know what your view on that is, Johannes.

Ioannis 14:15

Well, uh yes, there are two two two types of models, the fixed and the random effects. The the fixed effects model uh will provide one true effect uh size, uh while on the other hand, uh the random effects uh model uh will provide them we we apply this where we when we have studies that uh estimate a distribution a distribution of uh effects. Um but in uh medical uh sciences especially, uh it is most likely to use random effects because of the uh diverse nature of clinical uh protocols, for example, uh different populations, different uh dose regimes, uh different follow-up uh periods. So these all add uh to the heterogeneity of the study.

Heterogeneity And Subgroup Analysis

Matt 15:06

Yeah, so actually that that probably brings us on nicely to another sort of concept that you just brought up there heterogeneity. So this is something that I think some students get a little bit worried about and they don't really need to be too worried about it. So heterogeneity is a description of how similar or different the paper's results are compared to each other. So there are a couple of ways of measuring this. So the most common one that we see probably is the I squared statistic, which ranges from 0% to 100%, where 0% means all of your papers are in a fully agreement with each other, and 100% essentially means none of your papers are in agreement with each other. They may still all be indicating the same overall result, but the numbers are different. And sometimes students get you know an I squared value of 99, 100%, and they start to panic and think, what have I done wrong? But actually, really, this is just a description of the different papers. And the way I see it, I explain to students, is that it's an invitation to explore why this may happen. So perhaps we could have a little talk about what possible causes there are and what we might do about it if we find something.

Ioannis 16:12

Well, uh a major misconception uh for students uh is that uh, for example, no heterogeneity means perfect evidence. And the simple answer is of course no, because for example, all studies could share the same bias. Or uh another misconception uh is that uh uh an I square uh uh which uh uh measures heterogeneity uh above 50 percent uh means uh that the random effects is uh is wrong. No, because random effects uh is for presence of heterogeneity, not the severity of uh of uh uh heterogeneity. And of course, not uh uh uh in general, based on the Cochrane uh handbook for systematic reviews, when we have uh a high heterogeneity, uh we should apply the random effects model, while on the other hand, when we have a low uh heterogeneity, we need to uh apply the fixed uh effect uh effects models. This is a general uh rule where uh uh it is acceptable uh for publications, uh for example. Uh but uh in most of the cases, uh if uh we we start with uh a random effects uh model uh dependently of the heterogeneity, uh it uh it is not uh uh wrong. However, it uh it uh adds an uh uh unnecessary arm of uh for the study, for the meta-analysis, but uh it is general not wrong.

Shirley 18:06

Yeah, and I think often if um I think it's one of those things that we often talk to students about in terms of statistical results in general, isn't it? If they don't get a significant result, is what what happens in in, you know, is that wrong? But actually um there's always something you can talk about in the discussion, isn't there? And if you get high heterogeneity, it's something to kind of look into, isn't it? Is there something about the studies that might be different? Uh and sometimes that can lead students to do subgroup analysis. So uh I've seen this where perhaps students where studies have perhaps used slightly different dosages, for example, and maybe they can split their studies into two groups, one with a higher dose and one with a lower dose, and actually look at what the metro analysis shows them for each of those subgroups. Um and you can get obviously the results that you get for each subgroup, and then there will also be tests of how how different the subgroups are. Um, and I'll come on, I guess, to some of the output you get from metro analysis in a moment. But I think I think that sometimes there might not be any explanation for why there's high heterogeneity, why there's very different results, but actually there could be, and it's always worth looking, isn't there? Is there something different about the studies like the patients, the age of the patients, the gender, or the the region, geographical region that the study was carried out in? Um these are all things you can talk about in your discussion as possible reasons for why studies may have got differing results.

Ioannis 19:38

Or the genetic background in in some diseases.

Shirley 19:41

Yes, you're right. Sometimes it can be, and or perhaps the baseline measurements of the patients under particular measure.

Forest Plots And Interpreting Results

Shirley 19:48

So perhaps we ought to talk about some of the output that you get from metro-analysis.

Matt 19:53

So the the classic, I suppose, is your forest plot, which is a visual representation of the results from each paper and the overall pooled result. Um, and it will probably also include some statistics if you're using uh software such as Revman, it will be included in the statistics of that as well. And essentially what this tells you is just at a quick glance, what is happening in each paper and what is happening overall. So typically this will be expressed with an overall result as a diamond shape, and depending on which side of the plot that diamond is and whether it uh crosses over the what I call the line of no effect, uh, which is where both studies, so both arms of the study would be equal to each other. If a result crosses over that line, then it is essentially indicating that the two interventions that we're looking at aren't different from each other, which may be a good result depending on what you are trying to look at. So that's the classic forest plot. Um and there's a lot that you can get from that, but essentially it's just a quick visualization of the numbers.

Shirley 21:01

Yeah, and usually the software will give you the statistics that underpin that as well, won't it? The overall um figures, the overall summary statistics, but also a test of overall effect that says is it a significant difference or not? As you say, if it crosses the line of no effect, you know there's no effect. But if it's one side or the other, you you should see that those statistics show a significant result. There's a significant difference between um the two treatments or the the different interventions.

Matt 21:31

Which in some cases, finding a non-stagment result is actually what you want to find. So if you're looking at a non-inferiority trial, you just want to know is it not worse? On the other hand, if you're looking at the safety outcomes, uh you don't want it to be worse than um an alternative treatment. Or potentially, if you're looking at a new treatment which is significantly cheaper than an existing one, maybe you just want to know okay, is it as good as the existing treatment? Less money. So there's a few reasons why a non-significant result can actually be a good thing. And it really depends on the context of what you're actually looking at.

Ioannis 22:09

And at the end of the day, a non-significant statistically significant result is a result and should be reported. And uh sometimes uh not sometimes, in most of the cases, saves times time for uh from other researchers to to uh investigate this uh specific uh clinical issue.

Matt 22:32

Yeah, absolutely. That's um that's a really big point that I sometimes bring up with students is that we don't necessarily have uh a quantification of how much time has been wasted by researchers repeating results that don't work because it's not published. So we don't know how many times studies have been done and they found nothing and then not reported it. So it's it's really good practice to report all null results.

Funnel Plots And Missing Evidence

Matt 22:58

Um, and that actually brings on to one of the other things that you can get from meta-analysis is some visualizations of asymmetry. So we sometimes use a funnel plot to maybe tease out whether or not there are some results missing from the body of evidence that exists in the world, and that can sometimes be evidence that somebody has not reported some results, but you can't prove that.

Ioannis 23:24

Well, the the uh there is a clear guidance in the Koch Ray handbook for systematic reviews where when you have uh ten or more studies for one specific outcome, you should proceed and uh do the final plot to show a symmetry. Uh whereas when you have less than 10, you should not send it. Yes, that's another question that we see. Sorry.

Shirley 23:49

Yes, the final plot is less reliable, isn't it? If you've only got a few studies, because you you can't really assess that symmetry, particularly if you've only got three studies or something.

Matt 24:00

Yeah, yeah, in that case, with three studies, you will always have an asymmetry where two are on one side and one is on the other. There's always going to be that asymmetry. But that is something that we do get asked about sometimes is um, should I be doing a full pod? And as as we've just gone through. If you've got more than 10, yeah, go ahead.

Shirley 24:17

Yeah, so um with this talk of statistics, I I think it's good um perhaps for us to briefly touch on how you actually do a meta-analysis.

Software Options And Workflow

Shirley 24:26

And um, Ioannis I, you mentioned before Revman, um, and I think Revman is um really produces the nicest plots.

Ioannis 24:36

But of course it has its prons and cons uh because uh you can only do uh two-arm analysis uh with Revman and uh you can uh have a limited uh modeling. The other software is uh uh the free software Jamovi, uh where uh it is uh user-friendly, uh but sometimes it's uh less reliable for advanced meta-analysis.

Shirley 25:06

We uh certainly um we have resources to help students use Revman, um, and we can also help uh students um to use Jamovi.

Matt 25:16

Yeah, I mean I I often say to students, once they've extracted their data from papers, they've done the hard work really. Uh it's simply a case of putting it into the software. So we would generally recommend Revman. And that process, there's a few steps to it, but there's nothing that's difficult. So long as you have collected the data and extracted the right data, it's a very straightforward process. So that's absolutely something that we can help with in the academic skills team. Uh, we do this a lot. Um, so we can help the people um once they've extracted the data. Um, but say once you've once you've got the data, it's quite straightforward.

Shirley 25:52

So I I wonder if there's um yeah, anything else that we want to mention about meta-analysis.

Mixing Study Designs And Converting Data

Shirley 25:59

We it feels like we've covered quite a bit so far.

Matt 26:02

One thing that I sometimes see students saying is, oh, can I use uh randomized control trials along with longitudinal or observational trials? So what would your perspective on that be?

Ioannis 26:13

Uh well, from my point of view, yes, in in a single uh systematic review. Uh but uh these should be separated uh when uh you are uh uh analyzing the data either qualitatively or uh especially uh quantitatively.

Matt 26:30

Yeah, because I think that's something that we I've certainly seen students having some RCTs and some observational trials and wanting to try and combine them in a meta-analysis. So just to clarify, that's not really a good idea. Another thing we sometimes see is um papers where it's a continuous outcome and some of the papers have got means and standard deviation or something similar. And then perhaps one or two papers have got median and it quartile range or something similar. And sometimes it's got how I can combine these things. So there are a few procedures that we can do for this. So perhaps we could go through a couple of those.

Shirley 27:05

So I think Redman's quite good, isn't it, if you've got things perhaps around means of standard deviations or confidence intervals, you can actually plug in the values you've got, and it has a little calculator that will help you do that. I know I had a student recently who had uh means and quartiles reported for a lot of the papers and wanting to use software that obviously expects means and standard deviations. And there are a few um academic papers out there that have various methods for estimating means and standard deviations from medians and quartiles. Of course, it's slightly less reliable. And I think you mentioned before, Matt, we one of the first ports of call is probably going to the supplementary information, isn't it, to see whether actually they've got that information. But yeah, there are one or two methods out there for perhaps converting what you've got into what you need to use the software.

Ioannis 28:03

And we are very grateful to our fellow mathematicians who uh help us to convert uh one thing uh to another.

Matt 28:12

Yeah, yeah, some of those formula are not the nicest. So sometimes it can be nice to try and find an online calculator to do it. Although this is again something that if you do need to convert it by hand, uh academic skills can help you do those calculations.

Shirley 28:27

Yes, and I think it's just it obviously it's something to mention in your methodology if you've had to do those estimations, um, because it it may make your results slightly less reliable than if you've got actual means and standard deviations, for example. But but you know, again, it's something that you can work into um your write-up and discuss potentially the fact that it might be less reliable.

Closing Thanks And Student Support

Shirley 28:51

I hope you found this uh episode on meta-analysis helpful. I want to thank Johannes for joining us for this special episode. So uh thanks for listening and we hope you listen to the Skills Pod again soon.

Ioannis 29:03

Thanks everyone. I hope you find this uh podcast uh helpful. Thank you very much, everybody.

Emma 29:08

Hi there. If you're a University of Chester student, here are the ways you can access support from your academic skills team.

Anthony 29:15

On our Moodle pages, we've got lots of interactive resources for you to use. On our Literacies Moodle page, you'll find help with a range of skills from academic rating to revision. On our Maths and Statistics Moodle pages, you'll find help with different statistical tests, calculations, and formulas.

Emma 29:32

You can also use our feed forward email assistance service. You can send 750 words, which is around three paragraphs, of your work, to ask at chester.ace.uk and we'll respond within three working days with generic and developmental advice on aspects such as paragraph structure, criticality and referencing.

Anthony 29:52

You can also book a one-to-one with the Academic Skills Advisor via our Moodle pages. These appointments typically last 30 minutes and are available online and in person. Be able to see the campuses we're at by looking at our booking scheduler. You can send across an extract of your work for us to look at in preparation for the one-to-one. Or you can book a one-to-one to discuss a generic skill such as referencing or critical thinking.

Emma 30:16

If you and a group of your course mates are struggling with the same academic skill, you can book an Ask Together session by emailing ask at chester.ac.uk with details of your availability, how many people are in your group, what skills you want to cover, and where you'd like the session to take place.

Anthony 30:34

You can follow us on Instagram and Facebook using the handle AkETSkillsUOC, where we post practical tips on a range of academic skills, and it's also a great way to see what the team are up to.

Emma 30:46

And of course, you've got the skills pod. If you have a topic that you'd like us to cover or you'd like to be involved with our podcast, please email ask at chester.ac.uk.

Anthony 30:56

Ask.

Speaker 1 30:57

Supporting your success.