“Probiotic” therapy-in contrast to antibiotic therapy-defines the use of single bacterial strains or combinations of bacterial strains to influence the commensal gut microbiota, despite the fact that the term “probiotic” was not used before the 1950s.^1,2 Parker³ first established a definition in 1974. Fuller⁴ defined probiotic as a “living microbial food supplement” that improves the host by improving its intestinal bacterial balance. His definition is still valid today. This also resembles the position of the WHO/FAS (2001) which state that probiotics are living organisms that, when ingested in sufficient amounts, may be beneficial for the host.⁵ This definition avoids any speculation about the presumed mechanism of probiotics’ action. A 2014 update of the definition by the International Scientific Association for Probiotics and Prebiotics (ISAPP) confirmed⁶ the latter definition. In addition, bacterial yoghurt cultures usually consisting of Lactobacillus delbrueckii subsp. bulgaricus and Streptococcus salivarius subsp. thermophilus, which were not previously regarded as probiotics, are now included in this group.

The use of probiotics in the treatment of gastrointestinal (GI) tract disorders increased during the 1990s. Among the disorders that have been shown to benefit from probiotic treatment in placebo-controlled double-blinded studies are antibiotic-associated diarrhea,⁷ traveller’s diarrhea,⁸ rotavirus induced diarrhea,⁹ and Clostridium difficile-associated diarrhea.¹⁰ Beneficial effects have also been shown in diarrhea in children.¹¹ Other bacterial strains showed efficacy in chronic constipation.¹² Another potential indication was reported recently in the remission phase of ulcerative colitis,¹³ whereas randomized clinical trials (RCTs) in Crohn’s disease mostly revealed negative outcomes.¹⁴ Testing of probiotic therapy and targeting functional GI symptoms such as defecation problems (diarrhea and constipation) which are, however, not the predominant feature of diseases such as irritable bowel syndrome (IBS) were a significant step forward.

However, clinical trials prior to the formation of the international Rome consensus for functional bowel disorders in 1988 mostly lack credibility because a homogenous disease definition was lacking.¹⁵ The consequence was that in some trials, patients with or without abdominal pain in association with altered bowel habits were classified as IBS sufferers (or irritable colon, mucous colitis, or otherwise).¹⁶ In studies based on the Rome consensus IBS definition,¹⁷ efficacy was often based on improvement of individual symptoms including abdominal pain, diarrhea, and/or constipation.

A number of systematic reviews and meta-analyses^18-26 analysed the efficacy of probiotics in the management of IBS and these publications are frequently mentioned and cited in consensus papers.²⁷

Over the years general assessments have remained rather similar, presuming an overall positive effect whilst claiming a lack of high-quality data: “Probiotics may improve symptoms of irritable bowel syndrome and can be used as supplement to standard therapy.”¹⁹ “While our analyses suggest that probiotic use may be associated with improvement in IBS symptoms compared to placebo, these results should be interpreted with caution given the methodological limitations of contributing studies. Probiotics warrant further study as a potential therapy for IBS”²⁰: “Probiotics may have a role in alleviating some of the symptoms of IBS; however further research should focus on the type, optimal dose of probiotics and the subgroups of patients who are likely to benefit the most”²³: “Six of the eight diseases (...irritable bowel syndrome...) showed positive significant effects.... Across all diseases and probiotic species, positive significant effects of probiotics were observed for all age groups, single vs multiple species, and treatment lengths.”²⁵

Among these meta-analyses, the largest (including 35 trials) found a significant effect of probiotics on global IBS symptom rating for all 24 mono-strain preparations pooled²⁶; however neither lactobacilli nor bifidobacteria was effective when tested separately. There was also an overall benefit of the 15 combination trial; however, when specific combinations were tested, neither yielded a significant benefit over placebo. A significant effect of probiotics on individual IBS symptoms was only found for bloating and flatulence. A later meta-analysis covering data published between 2007 and 2013, which included only 14 RCTs, found probiotics to have a positive effect on abdominal pain only.²⁸

The initial idea of our project was a meta-analysis on the efficacy of probiotics in the management of IBS. However, after the initial search and data extraction (see Methods below), we concluded that the current data did not allow a methodologically sound meta-analysis to be performed. We therefore performed a new literature search with a changed scope: to critically review the body of evidence and to identify and assess the reasons for poor evidence in the current literature, as reflected in recent meta-analyses. We also proposed ways to improve methods for studies in the future. Our new analysis was conducted in 4-steps: (1) literature search, (2) identification of systematic reviews and meta-analyses, (3) comparison of lists from Step 1 and 2, and lastly (4) sorting the final list according to specific criteria.

In our initial search early in 2014, we performed a PubMed literature search using the search terms “(probiotic OR prebiotic) AND (irritable-bowel-syndrome OR IBS OR functional-bowel-disorder) AND trial” to retrieve randomized trials comparing a probiotic or prebiotic to a control treatment in functional bowel disorders such as IBS.

The abstracts of retrieved papers were screened to match the following inclusion criteria:

(1) The study including IBS patients according to pre-defined criteria (Manning, Kruis, and Rome);

(2) The study only including adult patients;

(3) The study design including a placebo arm;

(4) The study being described as randomized;

(5) The study with attempted blinding of patient assignment;

(6) The study describing the prebiotic or the probiotic bacterial strain in sufficient detail;

(7) The study assessing either global IBS measures or single symptoms or QOL;

(8) The full paper being available and written in English.

If only an abstract was available, we excluded the study. After exclusion of non-relevant studies, the remaining articles were screened for the following criteria:

(1) Whether the study population was IBS, IBS-C or IBS-D, and IBS-M;

(2) The number of patients included in the study;

(3) Whether the probiotic was single-strain or multiple-strain;

(4) In the case of multi-strain products, the strains included;

(5) Whether it was a pharmacological or nutritional probiotic;

(6) Whether the design was a parallel-group design or a cross-over design;

(7) The duration of treatment.

All screenings were performed by 2 authors (E.B. and M.S.). In case of disagreement, a third author (N.M.) was consulted for a final decision. We also assessed the Jadad score for each selected study in order to assess the methodological quality of the clinical trials.

After the first review of the retrieved data and the decision to change the purpose and methodology of the project, we implemented a 4-step approach that helped us achieve our goals.

In Step 1 we performed a new literature search using the same key words- “(probiotic OR prebiotic) AND (irritable-bowel-syndrome OR IBS OR functional-bowel-disorder) AND trial” -which resulted in the generation of new literature set. The last update was performed on April 20, 2015. Retrieved papers were screened again using the inclusion and exclusion criteria described above.

In Step 2 we identified systematic reviews and meta-analyses in our literature search published between 2006 and 2015, which summarise the respective actual state of knowledge. They were reviewed for the included studies as well their major findings across all studies. A complete list of trials included in the reviewed meta-analyses and systematic reviews was generated independently from the literature list from Step 1.

In Step 3 we compared both retrieved literature lists from Step 1 and from Step 2 and identified papers included in one or both lists as well as those that were excluded by us, but which were included in one or more of the performed meta-analyses. For each discrepancy found, we discussed the respective paper among all authors to reach a final decision about it’s inclusion or exclusion using the same criteria described in Step 1.

Step 4: the final list of included studies was sorted according to the following criteria:

(1) Single-strain or multi-strain probiotic;

(2) Manufacturer of the preparation;

(3) If multi-strain, which bacterial strains were present;

(4) The dosage of each strain;

(5) The form of preparation either as a nutritional product or a drug;

(6) If single-strain, the bacterial strain used, and at which dosage;

(7) Trial length and patient groups included;

(8) Study outcome as defined by the authors-positive or negative.

Based on our critical review of the current literature, we finally discussed the potential cornerstones of studies to investigate the efficacy of probiotics in IBS in the future.

Step 1: Figure 1 shows the results of the literature search, last performed on April 20, 2015. After exclusion of systematic reviews and meta-analyses (n = 9), uncontrolled trials (n = 20), narrative reviews, editorial, letters and comments (n = 78), animal research (n = 2), studies of children and respective reviews (n = 10), non-English papers and studies (n = 8), and others (n = 8), 60 studies remained. Data published in abstract form only was not included.

Following a full-paper review, we excluded 8 further studies due to various reasons: only 1 study reported data from prebiotic treatment,²⁹ 1 study did not use a probiotic in a strict sense (no living bacteria),³⁰ 2 were microbial investigations of earlier studies,^31,32 2 were single-blinded,^33,34 and 2 were uncontrolled interventions.^35,36 Based on hand-searches of the published meta-analyses, we added 3 more studies,^37-39 resulting in a total of 55 articles.

Step 2: we identified 10 meta-analyses and systematic reviews (published between 2008 and 2015) related to probiotic trials in IBS,^18-26,28 reporting a total of 44 RCTs with variable number of trials in each of them that increased over the years. One “systematic review” of complementary and alternative therapies in IBS was not comprehensive at all for IBS probiotic trials¹⁸; it included only 4 studies that were also analysed and discussed in all other reviews. Another review²¹ summarized previous systematic reviews and meta-analyses,^{19,22-24,40,41} but did not provide the full list of citations of papers and was therefore of no further help for the purpose of this review.

Step 3: we then compared the literature lists from Step 1 and Step 2 to identify the degree of overlap between each analysis. Of the 45 papers listed in Step 2, we excluded 14 for different reasons.

Six citations were available as abstracts only^42-47 and were thus of no value for our study. One of those⁴³ has been published as a full paper later⁴⁸ and was included in our sample; another⁴⁴ was also published as full paper³⁰ but was excluded since it was not a probiotic in a strict sense; 2^45,46 had only an English abstract however the full papers were written in Korean and Chinese each.

Two frequently included papers reported studies in children.^49,50 Two were placebo controlled but single-blinded studies,^33,51 one combined acupuncture with probiotic treatment,⁴⁶ and one³² reported microbiota changes from a probiotic trial published before⁵² which was already included in our list.

One study was not able to be verified⁵³ but was, by title, identical to another article by the same authors published in French⁵⁴ It was excluded because it was published too long before the first standardized diagnostic criteria for IBS to become available.⁵⁵

One more study⁵⁶ from Step 2 was added to our list of Step 1 studies, resulting in 56 studies available for our analysis.

Taking into account only the studies on our list and in at least one of the other analyses, overall agreement was low, as can be seen in Figure 2: only 5 RCTs from all 56 were included in all 7 previous analyses (unless the analyses restricted their timeframe, as in²⁸) as well as in ours. Seven RCTs were exclusively analysed by the largest study²⁶ and our analysis added another 20 studies to the list.

Step 4: of the 56 RCTs, 27 reported data from multi-strain probiotics, and 29 from single-strain probiotic products. Such a distinction was not made in the majority of the meta-analyses. Except in 2 studies,^24,26 all others pooled data and provided overall statistics and relative risk calculations.^{19,20,23,25,28} One²² did perform a systematic review instead of a meta-analysis, but nevertheless mixed both groups.

Multi-strain Probiotics

The 27 included studies using multi-strain probiotics are listed in Table 1.^{38,39,48,57-80} Despite this common character, the studies were heterogeneous in many other aspects.

Single-strain Probiotics

Our systematic review of 56 published RCTs of probiotics in IBS and of 10 systematic reviews and meta-analyses, including most of these trials, identified major flaws in the RCTs, obscuring the evidence for the efficacy of probiotics as a treatment option in IBS. Consequently, less evidence was produced with increasing numbers of RCTs added to meta-analyses. The latest and largest of these meta-analyses²⁶ concluded that while across all studies moderate evidence exists for efficacy of probiotics in general (though restricted to single-strain preparations) on global symptoms, neither individual bacterial strain reached a sufficient level of evidence, nor are individual core IBS symptoms (except bloating) effectively treated by any of the tested bacteria. Multistrain probiotics appear to be of no value at all. Narrative reviews on the efficacy of probiotic treatment in IBS usually follow these systematic reviews and meta-analyses.¹¹¹

In an attempt to understand why the picture has not improved but has rather developed into the opposite in recent years, we analysed all available studies for homogeneity in the applied drugs (or food supplements), in design and trial features and global outcomes and found that heterogeneity rather than homogeneity has increased over the years. Patient recruitment and selection especially, treatment duration, probiotic dosages and the choice of primary and secondary endpoints of the study have not followed the same route that pharmacology of GI drugs have paved through the FDA,¹¹² EMA,¹¹³ and international consensus parties such as the Rome group.¹⁷ Whilst in GI pharmacology placebo rates have dropped and efficacy of drugs over placebo has increased,¹¹⁴ the development of probiotics has not followed the same trend. One reason for this might be that, investments in clinical research and trials are still the exception rather than the rule for the nutritional (dairy) industry, not least because of lower profit rates and higher turn-around of investments into marketing of products.

Instead of rigorously questioning the value of previously tested probiotics in IBS, we finally attempted to outline the requirements for future trials to overcome these limitations. Many of these are already clinical standards that have been set during the last 25 years in GI drug development, by drug companies, approval authorities, and expert consensus parties. These have not been adopted yet by the nutritional industry that is responsible for most of the compounds that have been put to the market. However, clearly there cannot be different standards for drug and nutrient testing, and the future common policy of EMA and European Food Safety Authority in Europe¹¹⁵ has underlined this.

How to Study Efficacy of Probiotics in Irritable Bowel Syndrome

Since GI health claims for probiotics have to assess efficacy with respect to patients with functional bowel disorders of IBS-type,¹¹⁵ they have to be measured with standards of drug testing in the same area, ie, according to FDA and EMA rules^112,113 and the consensus of the gastroenterology community.¹⁷ The following provides a summary of putative paradigms to guide future trials.

First and foremost, any clinical study should be registered and it should be registered before the study start and not after the data evaluation.^57,86,92 Only 6 of the 27 multi-strain probiotic studies were registered. Even more surprisingly, only 3 of the 29 studies with single-strain probiotics were registered despite the fact that the single-strain probiotics were used not as nutritional supplement, but as drug by the majority. Registration prevents (silent) deviations from the proposed protocol and statistical evaluation strategy and diminishes doubts as to whether the reported efficacy data was a re-interpretation of missing efficacy on the primary endpoints chosen (see above).

Secondly, the study would need to be adequately powered. As we have repeatedly discussed,¹¹⁴ studies with samples size lower than 100 patients are at increased risk of producing high and variable placebo response rates and therefore would need strong effects on the primary study endpoint to achieve significance. However, this cannot be expected with nutritional interventions (even when provided as approved drugs) which is confounded by the variability of daily nutrition including consumption of preand probiotics.¹¹⁶ In this respect, 19 of the 27 multi-strain and 21 of the 29 single-strain probiotic studies had a priori low chance of finding significant effects; however, to our knowledge only a few were based on a power calculation.

Thirdly, a probiotic study in IBS patients would preferentially not use a cross-over design; such designs have almost completely been abandoned in gastrointestinal pharmacology and elsewhere for obvious reasons. They carry the risk of unblinding,¹⁰⁹ of cross-over effects when wash-out periods are too short,¹¹⁶ and of conditioning effects in the second (placebo) phase when the initial treatment is the probiotic.¹¹⁷ Consequently, the statistical evaluation should be restricted to the first treatment phase; however, these studies are usually not powered for such “quasi parallel design.” Only 4 of the single-strain studies and none of the others installed a cross-over design.

Furthermore, inclusion and exclusion criteria should closely follow the EMA and FDA guidelines for clinical trials in IBS,^112,113 including the definition of minimal severity for inclusion, global primary endpoints, and adequate secondary endpoints (pain, bloating, and a clinically meaningful responder definition). It should incorporate at least 8 weeks of treatment, an adequate follow-up interval and restriction to one of the different IBS subtypes. Documentation of nutritional habits as well as stool and pain diaries should be mandatory.

Finally, either a single-strain probiotic should be tested against placebo, or 2 different probiotics in a comparative effectiveness research design¹¹⁸ and the compounds chosen should be based on the available evidence. This excludes multi-strain probiotics, dairy products that require maintenance of the cooling chain and probiotics which have not been able to demonstrate superiority over placebo despite frequent trials, as documented in our list above. Comparative effectiveness research trials need larger samples (for demonstration of non-inferiority and the choice of the comparator is essential.¹¹⁸ A 3-arm trial (such as O’Mahony et al¹⁰⁴) testing 2 different probiotics against placebo is certainly an elegant alternative. Comparing probiotic treatment to other dietary or non-dietary therapies (prebiotics⁸⁶ and simethi-cone⁸⁷) or adding probiotics to other treatments (acupuncture⁴⁶) may help in recruiting patients but does not support the search for evidence.

We are aware of the limitations of our own analysis. While we included as many published trials as possible into this systematic review, we excluded trials reported as abstracts only and those not published in English. For a meta-analysis of overall efficacy, this may create a publication bias: however, for a critical review of methodological flaws this may not be as relevant since detailed data needed for critical analyses is usually not included into congress abstracts. We also ignored differences in primary endpoints (and their statistical characteristics, continuous or binary) reported in these studies and referred to reported overall efficacy data as “true” -assuming that the authors have reported the best evidence they could gain from their own data, irrespective of what the initial evaluation plan was. Otherwise, the overall evidence may have been even poorer. Finally, since most of the multi-strain preparations were nutritional supplements from different suppliers worldwide, the same product may have been available in different countries under different brand names with different labelling and approved for different indications due to differences in national regulations for approval of nutritional supplements. We were not able to resolve this issue.

When evaluating the efficacy of the FODMAP diet in IBS on behalf of the Rome Foundation, Yao et al¹¹⁶ thoroughly discussed the methodological difficulties of dietary trials, with an emphasis on functional bowel disorders. Most importantly, dietary intervention trials need to address the collinearity between food, nutrients, and bioactive components which could obscure the relationship between food and their effects in the gut. The authors provided recommendations for such trials, ranging from assessment and documentation of baseline nutrients intake, via adequate masking of study food, to the design of adequate sham controls; however, the authors insist that for food trials, the randomized, double-blind and placebo-controlled study remain as the gold standard.

In summary we conclude, that the heterogeneity of the studies of probiotics in IBS questions the value of meta-analyses. The use of different bacterial strains and different mixtures of these strains, as well as different dosages, are the main contributors to this heterogeneity. Current data provides limited evidence for the efficacy of a small number of single-strain probiotics in IBS (mostly bifidobacteria) and sound studies following strict trial guidelines (FDA and EMA guidelines for clinical trials) are needed.