Moving Away from WEIRD: Systems-Based Shifts in Research, Diagnosis, and Clinical Practice

In 2000, the World Association of Infant Mental Health (WAIMH) Handbook (Vol. 1) described the infant mental health (IMH) field as a “relatively young” … one engaged in a process of defining, evolving, and expanding itself (Fitzgerald & Barton, 2000, pg. 4). In this same seminal chapter, IMH was contextualized as being influenced by the theoretical forces of evolutionary, systems, and psychoanalytic theories. Each of these is implicitly a developmental theory that attempted to account for behavioral change over time. IMH was defined as holding multiple levels within its purview. This ranged from being an educational discipline, to a clinical field, to a research discipline. Collectively these levels coalesced around the social and emotional dynamics of infants and their caregivers. Important in this endeavor was the necessity of IMH being multidisciplinary and international, with the promise of emerging as an interdisciplinary field (Fitzgerald & Barton, 2000).

In this context, increasing concerns regarding cultural diversity and disparity were also emerging. It is important to note that within this same WAIMH chapter in 2000, it was recognized that a disproportionate amount of United Statesbased authors across all major scientific data bases existed. While there was some degree of diversity represented within the Infant Mental Health Journal (IMHJ) at this time, IMHJ declared a concerted effort towards shifting this disparity. In fact, the remainder of this WAIMH Handbook’s eleven chapters in Volume 1 (Fitzgerald & Osofsky, 2000), was dedicated to this very issue of demonstrating multicultural issues within infant mental health, representing 13 different countries. Staying true to its vision, and in sharp contrast to other scholarly and developmental journals involving infant research, the IMHJ showed an increase of 42% in its inclusion of population samples outside the United States or studies with infants of color from 2002 to 2006 (Fitzgerald, 2006).

Fifteen years later, we pause to reflect and survey the contextual landscape that surrounds the now “older and more mature” IMH field. We take a macro level viewpoint that looks at the continued influence of dynamic systems theory upon research and diagnosis; global concerns about cultural disparities from other lenses; and several National Institute of Health (NIH) and National Institute of Mental Health (NIMH) shifts (albeit from the United States) that reverberate and support the IMH field. Challenges to traditional research assumptions and practices with their influence on diagnosis and clinical practice have become even more prevalent in the last five to seven years.

In this paper we briefly review emergent concerns and how they have led to prominent shifts in the United States research community that may very well affect the international research community. These “top-down” NIH research shifts have short- and long-term implications for clinical practice, which continue to strive toward application to the real world populations being served. The drive from the “bottom-up” clinical community in-turn contributes to a bidirectional influence in moving research toward the same direction IMH has already claimed as its own – toward relational complexity. The field of IMH indeed continues to evolve and mature. As a field, we have been aware and a part of these shifts long before these have become national or world-wide movements. As a result, we are poised to participate in the current shifts in research and diagnosis unfolding across larger contexts with increased momentum. Our maturation will exponentially propel us further into new ventures of interdisciplinary collaboration.

Parallel Processes with IMH and Large Scale Research Movements

There are several large-scale concepts that the research community has acknowledged as concerns. We see this as a type of parallel process with many shifts the IMH field has already made. We will focus on the following shifts, adding more detail as the article progresses. 1) It has become increasingly evident to the larger research community that cultural and fiscal disparities reside with current population sampling strategies on a world-wide level. 2) It has become increasingly clear that that the use of diagnostic categories misalign with the clinical complexities of underlying systemic dimensions, dismantling the use of the Diagnostic and Statistical Manual of Mental Disorders (DSM) for research purposes and replacing it with NIH research domains since 2013. 3) It has become increasingly obvious to the NIH that there are limitations to linear research methodologies [even though systems theory has been around since the 1930’s (von Bertanlanffy, (1968); Sander (1987); Gottlieb (1991); Fitzgerald & Levine, 1992; Thelen & Smith (1994);Tronick, (2007), with many others, beyond the scope of this brief article, involved in the shift to systems approaches].

In recognition of the valuable application of systems theory to research, the NIH, Centers for Disease Control and Prevention, the Institute of Medicine, and many others (Mabry, et. al, 2013) have required adding nonlinear dynamic systems science methodologies to federal grants since 2009. It has become increasingly acknowledged that isolated research laboratories can no longer solve the complex problems we face. Hence the NIH Roadmaps emphasize the employment of translational, interdisciplinary, and community-based participatory research approaches.–All require an increase in collaborative efforts across disciplines and a shift from laboratory isolation to community involvement. As we review these items, we suggest that these are all part of an interdependent ecosystem comprising the “Parallel Process of Shifts in Research Perspectives that Enhance Clinical Practice” (see Figure 1). Similar to the parallel process model often applied to work in reflective supervision (Heller & Gilkerson, 2009; Heffron & Murch, 2010), this parallel process model proposes that movement or shifting within or across layers impacts and reflects the status of the other levels. The strength and viability of each layer is interdependent on and impacted by the variability, sensitivity, and responsiveness of their counterparts. As we explore the mismatches between current research and diagnostic approaches and their incongruence with the “real (clinical) world,” we hope to demonstrate that outside of the field of IMH there is a more recent push to move from the simple to the complex, from the linear to the nonlinear, from the categorical to the dimensional, from the laboratory (isolated) to the community (collaborative) –all embedded within a dynamic systems framework. We propose that as clinical theory, research, and practice in IMH has accepted a dynamic systems approach years ago, a growing parallel process has occurred within the Social and Behavioral research community on a larger scale. We are hopeful that as these “bottom-up” clinical and “top-down” research levels further promote dynamic systems theory, practice, and research on a much grander scale, that IMH, as an already existing interdisciplinary field, will make use of these shifts, coalescing into a stronger, wiser, more solid work force around the world.

Figure 1. Parallel Process of Shifts in Research Perspectives that Enhance Clinical Populations.

World Perspective of Mismatched Populations

The basis for most of the world’s top research journals comes from population samples drawn from “Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies” (Henrich, Heine, & Norenzayan, 2010). The inherent assumption in using this narrowly defined group is that this group is representative of the rest of the species, and research findings are generalizable to the rest of the world. However, in Henrich et al.’s (2010) review of comparable data bases across the behavioral sciences, the WEIRD subjects, rather than being representative of other world cultural groups, were actually rated as “frequent outliers” in many domains including essential features of motivation, psychology, and behavior, especially with reference to young children. Thus, what is often perceived as a “dominant” cultural group producing top-notch research that is applicable to all, is actually mismatched, not truly representative of the larger reality that exists from a global perspective. Representation of the “real world” population is further complicated by the historic distrust in research and health care existent in many underserved populations (Freimuth, Quinn, Thomas, Cole, Zook, & Duncan, 2001; Gamble, 1993). Additionally there is a mismatch between Euro-Western worldviews (upon which current academic research traditions are founded) and those of marginalized and oppressed groups that comprise groups who are served or who are in need of services (Chilisa, 2011).

As aforementioned in this article, the historical trajectory of research efforts in the field of infant mental health have considered and responded to the gaps potentially created by the WEIRD phenomenon (Fitzgerald, 2006). Therefore, we are positioned to join and contribute to current research efforts that encompass the real world population. However, as we move toward future research efforts, disparities still exist with non-Western, unindustrialized societies. Therefore, continued awareness of disparities existent between the representation of population samples and the real world population is essential.

Mismatched Populations in the WEIRD World

When we turn to concerns and challenges even from within this thin slice of WEIRD humanity, similar concerns about laboratory research being based upon mismatched populations have also been clearly voiced. Multiple types of mismatches have been documented between laboratory research settings and community practice. All the while, however, with these inequities in place, it is common for research laboratories to place their stamp of approval on the growing group of “evidence-based” treatments (EBT) documented as applicable to the “real-world,” even though they are often based on a population within the WEIRD world that does not match the at-risk and high-risk populations within any given community. These discrepancies will become more apparent as we continue the discussion below.

First, relatively simple, and often less complicated and less complex cases (e.g., singular diagnostic criteria that assumes a single causal pathway), that have been easiest to contact or those most compliant to come to hospital or laboratory based settings have been the source of both developing and testing “evidence-based medicine” and “evidencebased social science” treatments (Herbert, Brandt, Armstead, Adams, & Steck, 2009). In addition to this being somewhat tautological, the populations often studied for disease processes are often not representative of ethnic and economic minorities.

Similarly, from the social science side of research, infants and young children most often get into a particular study based on singular diagnostic categories, as most EBTs are diagnostic specific. The contradiction is that a singular diagnostic category is not the norm in community health/mental health settings. Rather, comorbidity across more than one diagnostic category, often increasing with degree of severity and family adversity, is the norm (Weisz & Gray, 2008). Not too long ago, an esteemed university in California came to a well-known community-based mental health clinic looking for singularly depressed 3-year olds. This clinic is located in South Central Los Angeles –zip codes holding a very at-risk population of infants and young children in Los Angeles. With high risk of co-morbidity and complex trauma, how many subjects could participate in this EBT study for depression? In reality, none.

In a more formal review of the literature, Weisz, Doss, and Hawley (2005) reviewed how research studies are conducted, identifying clinical versus non-clinical dimensions, and found the following concerns. Across dimensions of clinic settings versus research university settings, actual practicing clinicians versus graduate students or research-employed therapists, and actual treatment-seeking clients versus children recruited for efficacy trials, “only 1% of the studies reviewed included some clinically referred children, with at least one practicing clinician, with some treatment carried out in a clinical service setting” (Weisz, et al. 2005, p. 59, italics added). Thus, from this particular review, within the WEIRD and EBT worlds, 99% of research conducted is based upon 1% of an actual clinical population. In summary, EBTs are often based upon research done with singular variables (singular diagnosis), conducted by non-clinicians, and in nonclinical settings. What further complicates matters is that EBTs are also accompanied with, most often, a rather rigid set of manualized and prescribed interventions that have little respect for the time it might take to build rapport and a therapeutic alliance. Similarly, they rarely account for individual differences, nor accommodate spontaneous events that occur within the child or family system (Weisz & Gray, 2008). Subsequently, there is little room to tailor the EBT to the actual population being served.

Evidence-based practice, in its original definition from the Institute of Medicine, requires a three-pronged process of weighing and juggling the following variables: (a) the best of what research provides (this includes the EBTs); (b) the best of professional wisdom based upon theory and years of clinical practice; and (c) informed consent from parents, matching the family’s culture and needs (Sackett, et al, 1996; Buysee & Wesley, (2006); Brandt, Diel, Feder, & Lillas, 2012; Lillas, Feder, Diel, Brandt, 2014). In contrast, it is assumed that if one is using an EBT, that s/he is conducting an Evidence-Based Practice (EBP). These terms are often conflated and have become synonymous with each other. Thus, in many situations, EBP becomes a rote application of an “evidenced-based treatment” without the use of clinical wisdom that matches the neurodevelopmental needs of the children and parents with the type of EBT offered. Ideally, there would be an inclusive “menu” of treatments, with both bottomup and top-down neurodevelopmental applications that could be matched with the needs of the child, parents, and family system (Lillas & Turnbull, 2009).

The Parallel Process: Shifting from Simplicity to More Complexity

Building upon this inequity between research populations and clinical populations within the larger research world, we see three parallel processes that bring much needed transformation that will affect clinical practice. Essentially, these changes are dedicated to expanding to accommodate heterogeneity (individual differences) and complexity. The earlier discussion of restricting research studies to singular diagnostic categories and the associated challenges demonstrate the need for these shifts. For example, within early childhood, all too often we have the diagnosis of “autism” as the primary diagnosis defined in one categorical system (e.g. education, in United States legislation, Part C of the Individuals with Disabilities in Education Act) and “trauma” or “post-traumatic stress disorder” as the designated diagnosis for mental health providers. In many fiscal systems, the categories of developmental delays and mental health emerge from bi-furcated funding streams, which in turn support fragmentation and health care that is siloed. Even if the funding is not compartmentalized, the diagnostic categories often are, and these diagnostic “camps” operate without a common framework or shared approach. These are false dichotomies, and the dynamics reviewed here reveal more about these inaccuracies that are now being caught and challenged.

Parallel Process: Shifting from Categorical to Dimensional Perspectives Within Diagnoses

A few weeks before the long-awaited Diagnostic and Statistical Manual of Mental Disorders V (DSM) was released in May, 2013, a seismic shift occurred when the National Institutes of Health (NIH) and Mental Health (NIMH) both announced that they would no longer be funding research based upon DSM criteria (Insel, 2013). Whereas having a common language through the DSM or the International Statistical Classification of Diseases and Related Health Problems (ICD-10) has been useful, the criticisms are rooted in the fact that these manuals are organized around symptom-based categories that reflect an outdated view of how the mind, body, and relationships work. By analogy, they offer a symptom checklist approach to assigning diagnostic value. This is akin to practicing medicine by creating diagnostic categories and treatments based solely upon the symptom of a fever or the subjective type of chest pain, rather than referring to objective laboratory measures that now exist across many levels – such as genetics, molecular and cellular biology, imaging studies, neural circuitry, physiological activity, cognitive science, and other levels of information based in brain-behavioral relationships (Insel, 2013) and NIMH (nd), retrieved February 5, 2015).

The goal of this next decade is to collect new data based upon a framework of Research Domain Criteria (RDoC), beginning with five domains of functioning: (a)positive valence systems, (b) negative valence systems (e.g., positive and aversive motivational systems), (c) cognitive systems, (d) systems for social processes, and (e) arousal/modulatory/ regulatory systems. These domains are intended to be a starting point and are not set in stone. These realms can be then studied across different classes of variables –from genes, molecules, cells, neural circuits, physiology, behavior, and selfreport. The long-term goal is to transform how diagnoses are made, with much more specificity and complexity, and eventually, to offer the right type of treatment(s) matched with much more specificity and complexity. In order for transformation to occur, a change from current funding practices that recognize the DSM and ICD- 10 as the “gold standards” is essential.

In addition, the critique of the DSM process of categorization observes that many of the “same” symptoms can be found in multiple DSM diagnostic categories, and large categories, such as “depression”, carry very little specificity, lacking an awareness of individual differences. In this regard, there is a fundamental shift to looking at underlying dimensions that may cut across multiple DSM diagnostic categories. This dimensional shift is dedicated to looking at a full range of typical to atypical ranges of behavior. How clinical sample research populations will be garnered will also change to include a much wider range of constituents. Thus, a shift will occur away from the narrow requirement of having only a singular diagnosis in order to get into the study, and subjects will intentionally include more diagnostic complexity as well as “more clinical settings ” (Cuthbert & Kozak, 2013, p.930). For example, all clients across a range of “mood disorders” within a clinic setting may be studied rather than only those meeting criteria for a strict, single major depressive episode.

In the field of IMH, there is an ongoing urgency to reduce the disparities gap existent for low socioeconomic, rural, racial, and ethnic minority populations in the access to appropriate services. We are hopeful that shifts towards more specificity and complexity in the diagnostic realms will continue to address this issue with increased momentum. Similarly, research paradigms that address the complex and dynamic nature of this work are essential to the maintenance and continued growth of our field.

Parallel Process: Shifting From Linear to Non-linear (Systems Science) Research Methodologies

As specificity and complexity gather momentum in the diagnostic world of research, research methodologies that can hold complexity and specificity at the same time also gather momentum. Systems theory is an intellectual theory and research paradigm often associated in its origins with Austrian biologist Ludwig von Bertalanffy (1968). Clinical examples include family systems theory as well as incorporating an understanding of biological systems. While dynamic systems approaches have carried more weight in engineering, aeronautics, and biology, a slow and steady growth over the last few decades has evolved to include medicine, social and behavioral sciences, and public health (Mabry, et. al., 2013).

Part of this shift has come from a growing consensus that the social and behavioral sciences have relied on traditional linear research methodologies that typically narrow the problem, isolate variables, and use linear analytic representations. Linear models will continue to be useful and are necessary. However, it is increasingly recognized that research methodologies that hold more complexity, nonlinearity, interactive phenomena, bidirectional feedback loops, multiple causation, and time-delayed effects are essential in order to deal with more complex phenomena (Mabry & Kaplan, 2013; Mabry, Milstein, Abraido-Lanza, Livingood, & Allegrante, 2013).

In conjunction with this paradigm shift, technological advances have made systems modeling and simulation methods more accessible to the average research investigator. NIH has made significant investments in the use of systems methodologies for large-scale medical disease processes (e.g., cancer, obesity, diabetes), and these models are now moving into the social and behavioral arenas. Systems models are increasingly a part of the NIH federal grant requirements, with NIH’s investment in financing systems methodologies from 2009 to 2013 coming to a grand total of $35.4 million for 91 projects (Mabry & Kaplan, 2013). This is a significant investment surge “considering that prior to 2008 there were no NIH funding announcements for the behavioral and social sciences that were focused on systems science” (Mabry & Kaplan, 2013, p. 11S).

System methods aim to enhance understanding about real world systems, affording translation of evidence into practice, often through the generation of computational models. Three specific methods have been applied to real world problems in social and behavioral sciences, and are highlighted in the literature (Burke, et al., 2014). Systems dynamics modeling potentiates the capacity to frame, understand, and discuss challenges embedded in complex system. Agent based modeling employs computational methods to examine environments (e.g. communities) and agents (individual community members) situated within. In network analysis, the relationship between a set of nodes (e.g. community members and organizations) are examined and can be used to track relational changes in direction (unidirectional/bidirectional) and the “thickness” of social relationships within family and community settings.

In conclusion, systems thinking pushes us to replace narrow, reductionist, static, and short-term views of individuals, relationships, communities, diagnostic formulations of health/disease, and treatment interventions with broad, dynamic, and complex perspectives that take into account multiple considerations. Thereby, the complexities and interdependencies of the real world become incorporated into the research models.

The Parallel Process: Shifting to Translational and Interdisciplinary Perspectives

Several initiatives have attempted to reduce the gap between the promise of evidence based mental health practices and the realities of community practice. The NIH Roadmap for Medical Research (as cited in National Institute of Health, 2014) speaks to both the need for translational and interdisciplinary research. From one angle, NIH asserts that “in order to improve human health, scientific discoveries must be translated into practical application.

Such discoveries typically begin at “’the bench’” with basic research –in which scientists study disease at a molecular or cellular level-then progress to the clinical level, or the patient’s bed side” (as quoted in Herbert, et al., pp. 1214, 2009). When considering the application of translational research in IMH, this bench-to-bedside approach extends into whole communities and populations. In order to reap the benefits of scientific discoveries, findings must be available to all communities in a shared language. A research orientation such as this promotes a partnership between “researcher” and community “participants.”The partnership is undergirded by shared power, inclusivity, and the recognition of multiple forms of knowledge (Minkler & Wallerstien, 2010). Hence, research becomes a shared tool that empowers communities to explore solutions to mitigate disparities and gaps that exist in mental health care.

From another angle, and within a similar process of creating increased links, NIH describes health research as commonly being organized “much like a series of cottage industries, lumping researchers into broad areas of scientific interest and then grouping them into distinct, departmentally based specialties” (p. 1214, as cited in Herbert, Brandt, Armstead, Adams, & Steck, 2009). Recognizing the fragmentation of this siloed approach, NIH acknowledges that researchers will have to move beyond the confines of their own disciplines and foster shifts in academic culture to create collaborative models for team science (NIH Roadmaps, 2014).

Considered together, movement toward participatory-translational and interdisciplinary perspectives hold promise in significant reduction of disparities in health status existent between the WEIRD and real world populations. Further, the opportunity to work across disciplinary and academic /clinician silos is essential to unification of a currently fragmented system. Rather than working in isolation, we envision the adoption of a collaborative perspective (see Figure 1) in which researchers, clinicians, and communities operate in synchronicity. Shared knowledge and expertise is exchanged and co-created, then used as a tool to address the issues and challenges that exist within the lives of young children and their families. We now explore these orientations.

Translational research

Translational research is intended to “translate” or move basic research discoveries into practical application. Occurring across a continuum (DankwaMullan, et al., 2010), translational research is bidirectional, cyclical, and includes the following domains: (a) basic science discovery; (b) testing and application in developmental stages; (c) outreach and dissemination of findings, and (d) adoption and implementation. Although the domains appear in linear progression from discovery to dissemination, translational research harnesses the value of findings as they emerge (e.g. during outreach and dissemination) and allow their application to discovery and development.

Interdisciplinary research

In addition to the recognition of translational research model, the NIH advocates interdisciplinary approaches as a means to mitigate gaps existent between traditional approaches and “real communities” and to respond to the dynamic process inherent in the complex study of humans. Interdisciplinary approaches allow for intellectual exchanges across disciplinary boundaries. Shared expertise allows greater effectiveness in harnessing the collective power of individual strengths to solve “real problems.” Hence, we must move away from antiquated approaches that group researchers into distinct, departmental based specialties. Rather, it is incumbent upon researchers to keep pace with scientific discovery through implementation of an interdisciplinary lens. Current NIH initiatives support interdisciplinary training and the creation of specialized centers. Initiatives such as this support, with translational research, a shift toward decreasing the mismatch between traditional research practices and the real world. More specifically, research orientations that engage the expertise of community members as partners in addressing and understanding “real problems” suggest a compatible approach to minimize disparities and closing gaps. Community Based Participatory Research illustrates a research approach applicable to the creating meaningful and sustainable change and will be addressed further in upcoming Perspectives in IMH issues.

Recommendations

Mismatches and parallel processes co-occur across multiple layers and levels of clinical science and practice. We have covered a range of them –from an international perspective, to a US/ industrialized perspective –along with important shifts that offer some correction to these concerns. We inherently see these shifts as part of that double feedback loop process that is changing and challenging our existing mental models and frameworks, taking us further down the road into complexity and collaboration. We offer the following recommendations:

Be a responsible consumer of EBTs; notice the exact population, setting, and types of providers offering the EBT.
Become an advocate for families in communities being offered EBTs that have evolved from subclinical mismatches and disparities.
Even with EBTs targeted to actual highrisk clinic populations, not all were found to be superior in outcomes to “usual care” being offered. Do not disregard “usual care” as an anomaly but include it in an expanded data base and treat it with respect and as part of the informed choice families have (Weisz & Gray, 2008).
Support community-based orientations in research. Get involved on a ground floor level as a practitioner working in a community setting, working with real-world families that have multiple problems and co-morbidities.
Use frameworks that hold multiple causalities, cut across diagnostic categories, and shed light on underlying neurodevelopmental processes and dimensions.
Accept the challenge to work across silos. Move away from disciplinary fragmentation into interdisciplinary work –whether on an academic, research, or clinical level.

References

Brandt, K., Diel, J., Feder, J., & Lillas, C. (2012). A problem in our field: Making distinctions between evidence-based treatment and evidence-based practice as a decision-making process. Zero To Three, 32(4), 42-5.
Burke, J. G., Lich, K. H., Neal, J. W., Meissner, H. I., Yonas, M., & Mabry, P. L. (2014). Enhancing dissemination and implementation research using systems science methods. International Journal of Behavioral Medicine. Advanced online publication, DOI 10.1007/s12529-014-9417-3.
Buysee, V., & Wesley, P. W. (2006). Evidencebased practice in the early childhood field. Washington, DC: ZERO TO THREE.
Chilisa, B. (2011). Indigenous research methodologies. Thousand Oaks, CA: SAGE Publications.
Cuthbert, B. & Kozak, M. (2013). Constructing constructs for psychopathology: The NIMH research domain criteria. Journal of Abnormal Psychology, 122(3), 928-37. doi: 10.1037/a0034028
Dankwa-Mullan, I., Rhee, K. B., Stoff, D. M., Pohlhaus, J. R., Sy, F. S., Stinson Jr, N., & Ruffin, J. (2010). Moving toward paradigm-shifting research in health disparities through translational, transformational, and transdisciplinary approaches. American Journal of Public Health, 100(S1), S19-S24.
Delgado, S. V., Strawn, J. R., & Pedapati, E. V. (2015). Key pioneers of two-person relational psychology. Contemporary Psychodynamic Psychotherapy for Children and Adolescents (pp. 62-78), Berlin Heidelberg: Springer.
Fitzgerald, H. E. (2006). Cross cultural research during infancy: Methodological Considerations. Infant Mental Health Journal. Vol.27(6), 612- 617. DOI: 10.10021imhj.20111
Fitzgerald, H. E., & Barton, L. R. (2000). Infant Mental Health: Origins and Emergence of an Interdisciplinary Field. In J. D., Osofsky & H. E. Fitzgerald, (Eds). WAIMH Handbook of infant mental health, Vol 1: Perspectives on infant mental health (pp. 3-36), New York: John Wiley & Sons.
Fitzgerald, H. E., & Levine, R. L. (1992). Analysis of dynamic psychological systems (Vol. 2). New York: Springer.
Freimuth, V. S., Quinn, S. C., Thomas, S. B., Cole, G., Zook, E., & Duncan, T. (2001). African Americans’ views on research and the Tuskegee Syphilis Study. Social Science & Medicine, 52(5), 797-808.
Gamble, V. N. (1993). A legacy of distrust: African Americans and medical research. American Journal of Preventive Medicine, 9(6), 35-8.
Gottlieb, G. (1991). Experiential canalization of behavioral development: Theory. Developmental Psychology, 27(1), 4-13.
Heffron, M. C. & Murch, T. (2010). Reflective supervision and leadership in infant and early childhood programs. Washington, DC: ZERO TO THREE.
Heller, S. S., & Gilkerson, L. (Eds.). (2009). A practical guide to reflective supervision. Washington, DC: ZERO TO THREE.
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3), 61-83. doi: 10.1017/ S0140525X0999152X
Herbert, J., Brandt, H., Armstead, C., Adams, S., & Steck, S. (2009). Interdisciplinary, translational, and community-based participatory research: finding a common language to improve research. Cancer Epidemiological Biomarkers Preview, 18(4), 1213-1217.
Insel, T. (2013). National Institute of Health, Director’s Blog: Transforming Diagnosis. Retrieved from https://www.nimh.nih. gov/about/director/2013/transformingdiagnosis.shtml.
Lillas, C., Feder, J., Diel, J., & Brandt, K. (2014). Weighing the evidence: Evidence-based practice and evidence-based treatments in infant mental health. In K. Brandt, B. Perry, S. Seligman, & E. Tronick, (Eds). Infant and Early Childhood Mental Health: Core Concepts and Clinical Applications. Arlington, Virginia: American Psychiatric Publishing.
Lillas, C. & Turnbull, J. (2009). Infant/Child mental health, early intervention, and relationship-based therapies: A neurorelational framework for interdisciplinary practice. New York: W. W. Norton.
Mabry, P. L., & Kaplan, R. M. (2013). Systems science: A good investment for the public’s health. Health Education & Behavior, 40(1 suppl), 9S-12S.
Mabry, P. L., Milstein, B., Abraido-Lanza, A. F., Livingood, W. C., & Allegrante, J. P. (2013). Opening a window on systems science research in health promotion and public health. Health Education & Behavior, 40(1 suppl), 5S-8S. doi: 10.1177/1090198113503343.
Minkler, M., & Wallerstein, N. (Eds.). (2010). Community-based participatory research for health: From process to outcomes. Hoboken, NJ: John Wiley & Sons.
National Institute of Health (2014). NIH roadmap for medical research. Retrieved from https://www.thefreelibrary.com/ NIH+roadmap+for+medical+research.-a0115034508
National Institute of Mental Health (n.d.). NIMH Research Domain Criteria (RDoC). Retrieved from https://www.nimh.nih. gov/research-priorities/rdoc/nimhresearch-domain-criteria-rdoc.shtml.
Sackett, D. L., Rosenberg, W. M. C., MuirGray, J. A., Haynes, R. B., & Richardson, W. S. (1996). Evidence-based medicine: What it is and what is isn’t. British Medical Journal, 312, 71-2.
Sander, L. (1987). Awareness of inner experience: A systems perspective on self-regulatory processes in early development. Child Abuse & Neglect. 11(3), 339-346.
Sterman, J. D. (2006). Learning from evidence in a complex world. American Journal of Public Health, 96(3), 505-514. doi: 10.2105/ AJPH.2005.066043
Thelen E., & Smith L. B. (1994) A dynamical systems approach to the development of perception and action. Cambridge, MA: MIT Press.
Tronick, E. (2007). The neurobehavioral and social-emotional development of infants and children. New York: W. W. Norton.
von Bertalanffy, L. (1968). General system theory: Foundations, development, applications (Rev. ed. 1976). New York: G. Braziller.
Weisz, J. R., Doss, A.J., & Hawley, K.M. (2005). Youth psychotherapy outcome research: A review and critique of the evidence base. Annual Review of Psychology. 56:337-63. Doi: 10.1146/ annurev.psych.55.090902.141449
Weisz, J. R., & Gray, J. S. (2008). Evidence‐ based psychotherapy for children and adolescents: Data from the present and a model for the future. Child and Adolescent Mental Health, 13(2), 54-65. Doi: 10.1111/j.1475-3588.2007.00475.x

PDF