www.sciedupress.com/ijhe International Journal of Higher Education Vol. 6, No. 1; 2017 The Development, Validity, and Reliability of the Barriers to Seeking Psychological Help Scale for College Students Nursel Topkaya1, Ertuğrul Şahin2 & Betül Meydan3 1 Faculty of Education, Ondokuz, Mayıs University, Samsun, Turkey 2 Faculty of Education, Amasya University, Amasya, Turkey 3 Faculty of Education, Ege University, İzmir, Turkey Correspondence: Nursel Topkaya, Division of Guidance and Psychological Counseling, Faculty of Education, Ondokuz Mayıs University, Samsun, Turkey. Tel: 90-362-312-19-19-5368. Received: October 13, 2016 Accepted: November 3, 2016 Online Published: November 10, 2016 doi:10.5430/ijhe.v6n1p48 URL: http://dx.doi.org/10.5430/ijhe.v6n1p48 Abstract The aim of this study was to develop a scale to determine the barriers affecting psychological help-seeking in college students. In line with this purpose, the validity and reliability of the scale were examined in five different studies. Exploratory factor analysis results suggested that the scale consists of five dimensions, labeled as fear of being stigmatized by society, trust in the mental health professional, difficulties in self-disclosure, perceived devaluation and lack of knowledge. The confirmatory factor analysis cross-validated the scale’s five-factor structure on a new college students’ sample. The convergent and divergent validity of the scale were also supported with theoretically and empirically expected correlation with self-concealment, self-stigma, and attitudes towards seeking psychological help. Moreover, the scale was found to have an adequate stability coefficient across three weeks and appropriate levels of internal consistency coefficient reliability for each of the subscales in two different samples. These initial findings show that the Barriers to Seeking Psychological Help Scale is a valid and reliable instrument to measure barriers related to psychological help-seeking for college students. Future studies can examine the validity and reliability of the scale on different samples and cultures, as well as predictive validity on college students’ decisions to seek counseling. Keywords: Barriers to seeking psychological help, Validity, Reliability, Mental health, College students, Turkey 1. Introduction Previous research has shown that more than half of the mental disorders start before the age of 24 (Kessler et al., 2005, 2007; Wang et al., 2005). According to the results of a nationwide study, mental disorders are also common in both adolescents and adults in Turkish society (Erol, Kılıç, Ulusoy, Keçeci, & Şimşek, 1998). This is concerning, because previous research also revealed that when these mental disorders are not treated, they can result in undesirable psychological, social, and economic problems including inadequate educational gains (Mojtabai et al., 2015), low academic success (Keyes et al., 2012), dropping out of school (Melkevik, Nilsen, Evensen, Reneflot, & Mykletun, 2016), failures in social relationships (Idstad et al., 2015), substance abuse (Weitzman, 2004), and loss in the labor market (Rudolph & Eaton, 2015). Considering the prevalence of mental disorders worldwide, and the negative effects of these disorders if they are not treated, determining the barriers inhibiting people from utilizing mental health care services is of paramount importance for both maintaining community mental health and delivering effective mental health care services. Although mental disorders are prevalent and appear to be on the rise among college students (Hunt & Eisenberg, 2010), studies have consistently shown that college students underutilize mental health services for their academic and emotional problems (Topkaya, 2014a, 2014b, 2015a; Vogel, Gentile, & Kaplan, 2008; Vogel, Wade, & Hackler, 2008; Wang et al., 2005). Given that counseling has numerous benefits and not seeking counseling, on the other hand, brings many adverse consequences (Eisenberg, Golberstein, & Gollust, 2007). The early diagnosis and treatment of mental problems in emerging adults, a period that roughly spans the ages of 17 to 25 in Western countries, is of critical importance. In order to effectively direct people to receive mental health care services, the barriers inhibiting them from seeking help need to be determined. Thus, in recent years, some research has been conducted to determine the barriers influencing people’s willingness to seek psychological help when they are faced with intrapersonal and Published by Sciedu Press 48 ISSN 1927-6044 E-ISSN 1927-6052 www.sciedupress.com/ijhe International Journal of Higher Education Vol. 6, No. 1; 2017 interpersonal problems (Kuhl, Jarkon-Horlick, & Morrissey, 1997; Setiawan, 2006). Barriers related to psychological help seeking are defined as a general unwillingness to receive psychological help or factors hindering an individual’s psychological help seeking behavior (Kuhl et al., 1997). Kuhl et al. (1997) revealed that the most important factor inhibiting seeking psychological help is perceiving oneself, family, or friends as adequate in solving problems. In a study with high school students, Wilson and Deane (2012) examined the relationship between the intention to seek psychological help for personal, emotional, and suicidal problems, and the barriers to seeking psychological help in adolescents. Researchers found that students emphasized autonomy as the most influential inhibiting factor for themselves. Students believed that they were competent enough to solve their own problems without any professional help. Other important barriers identified in this study were feeling ashamed and restrained, fear of being misunderstood, being forced to do things they did not want to do, and issues surrounding time and money. In another study concerning the barriers related to seeking psychological help, Stanhope (2002) found that male students perceived more barriers than female students. Also, the barriers for students were categorized as the fear of being stigmatized, arranging an appropriate time for counseling, privacy concerns, doubts about the usefulness of the mental health professional, self-awareness, and feeling oneself competent. Lastly, Calloway (2008) concluded that while the barriers to receiving psychological help were the fear of being stigmatized, fears concerning treatment, and issues about self-disclosure and suicidal tendencies in undergraduate students, the most significant barriers was found to be the fear of being negatively judged by others. The majority of previous studies have attempted to identify the factors related to the attitudes towards seeking psychological help in Turkey. These studies most commonly investigated the sociodemographic correlates (e.g., gender, class level, perceived socio-economic status, parental education level, past counseling experience) in attitudes towards seeking psychological help (Keklik, 2009; Koydemir-Özden, 2010). Other researchers investigated the problems undergraduate students encounter and the psychological resources they use to cope with them (Kılınç & Granello, 2003), as well as undergraduate students’ willingness to seek psychological help and the associated factors (Erkan, Özbay, Cihangir-Çankaya, & Terzi, 2012). More recent studies have focused on the factors affecting college students' and adults’ willingness to seek psychological help (Topkaya, 2014a, 2014b, 2015a, 2015b). These studies revealed some facilitators and barriers related to psychological help-seeking. Such that being female, past counseling experience, positive attitudes towards help-seeking,the availability of psychological services, the belief in the benefits of psychological services, trusting in the mental health professional, and receiving psycological help free of charge is more likely to be facilitating factors in help-seeking process in Turkish people. On the other hand, self-stigma, public stigma, unwillingness to share problems with an unfamiliar persons, and self reliance is more likely to be barriers in help-seeking process. However, to date, we are not aware of any study focusing on barriers related to seeking psychological help in Turkey, except for a quantitative research study with undergraduate students (Koydemir, Erel, Yumurtacı, & Şahin, 2010). Specifically, Koydemir et al. (2010) examined students’ opinions concerning the barriers related to seeking psychological help, university adaptation, students’ needs, their psychological resources used, and their attitudes towards seeking psychological help. Following the interviews conducted in the context of study, the most commonly mentioned barriers to seeking psychological help was that students did not believe that they had any problem they could not solve on their own. In other words, they thought that they did not have a problem that required them to seek help from a professional. However, these people stated that they would consider seeking psychological help from a mental health professional in troubled times. The belief in being able to solve problems on their own, seeking help from close friends, and reluctant attitudes towards self-disclosure to a mental health professional were among the other reported barriers in the study. Taking into account the above studies that have begun to identify potential barriers to seeking treatment, the barriers to seeking psychological help can be grouped under three categories: personal barriers (e.g., personal features and situations), socio-cultural barriers (e.g., cultural values or characteristics of a specific community), and barriers related to institutions delivering help (e.g., psychological counseling services, psychological counselors, and other administrative issues) (Setiawan, 2006). Recent research demonstrated that there is a negative relationship between willingness to seek psychological help and perceived barriers (Setiawan, 2006). In other words, as the number of barriers people perceive increases, their willingness to seek psychological help tend to decrease. Considering the studies related to seeking psychological help in Turkey, there has not been any research focusing directly on the barriers to seeking psychological help. Additionally, there have not been any instruments developed that can be used to determine the barriers college students perceive, except for Mansfield, Addis, and Courtenay (2005),which focused on revealing male college students’ barriers related to seeking help about health problems. Two obvious main limitations exist in this instrument. Firstly, this scale developed in American male college students to determine Published by Sciedu Press 49 ISSN 1927-6044 E-ISSN 1927-6052 www.sciedupress.com/ijhe International Journal of Higher Education Vol. 6, No. 1; 2017 their help-seeking barriers in health related issues. Barriers related to psychological help may differ from other health related problems at least in magnitude and severity. Studies have shown that regardless of the specific psychiatric diagnosis or level of disability, people tend to stigmatize people with mental illness more severely than those people with other health problems (Corrigan et al., 2000; Weiner, Perry, & Magnusson, 1988). Secondly, male sample also makes it difficult to generalize to barriers other samples such as females. One of the most consistent results in the diverse area of help seeking, including academic work, counseling, physical and mental health is that females more likely to seek help to a greater degree than do male (Nam et al., 2010). Thus, identifying common barriers to help-seeking process is crucial for developing nuanced interventions for both sexes. The aim of this study is to develop a scale to determine the barriers affecting undergraduate students seeking psychological help and to examine its validity and reliability in college students. Investigating these barriers can contribute to revealing and eliminating the factors that prevent undergraduate students from receiving mental health care services. Also, determining these barriers is of importance in terms of planning appropriate interventions to support students’ mental health and well-being, and enhance mental health care services (Stanhope, 2002). 2. Method 2.1 Participants The Barriers to Seeking Psychological Help Scale (BSPHS) was developed through the five different studies. Participants were recruited using the convenience sampling method except for fourth study. The participants of first study was used to generate the item pool related to the scale and construct a draft scale; the participants of second study was employed to determine the number of factors to retain in the draft scale and exploratory factor analysis. The participants of third study was used for confirmatory factor analysis in order to determine whether the factor structure of the scale was similar in a different sample. The participants of fourth study consisted of randomly recruited participants from the second and third sample used to examine the BSPHS’s convergent and divergent validity. Lastly, the fifth sample was employed to determine the BSPHS’s test-retest reliability across three weeks. The participants of first study consisted of 353 students from various faculties and departments in a university in the Aegean region of Turkey. There were 171 (49%) male and 182 (51%) female participants. Of the 353 participants, 93 (26%) were in their first university year; 100 (28%) were in their second university year; 97 (27%) were in their third university year; and lastly, 68 (19%) were their fourth university year. The participants of second study similarly consisted of 205 students from various faculties and departments in a university in the Aegean region of Turkey. Of the 205 students, 77 (38%) were male and 128 (62%) were female. The age of the students ranged from 18 to 27, and the mean age was 21.00 (S.D.: 2.00). Most of the participants were first year students (38%), followed by second year students (32%), third year students (22%), and fourth year students (8%). There were 45 (22%) students who had previously sought psychological help. The participants of third study consisted of 222 students from various faculties and departments in a university in the Aegean region of Turkey. There were 87 (39%) male and 135 (61%) female participants. The participants’ age ranged from 18 to 39, and the average age was 21.32 (S.D.: 2.26). Of these participants, 72 (32%) were in their freshmen year, 62 (28%) were in sophomore year, 55 (25%) were in junior year, and 33 (15%) were in senior year. There were 54 (24%) students who had received psychological help. The participants of fourth study consisted of randomly selected participants from the second and third study. We give information students in the second and third study about that some questionnaires also included scales for another study that aims to determine the correlates of barriers to seeking psychological help before distributed the questionnaire. Students who voluntarily to answer these questionnaires composed of fourth sample. There were 245 (92 males and 153 females) students who ranged in age from 18 to 39, and the average age was 21.47 (S.D.: 2.35). The majority of these students were in their sophomore year (n=77, 31%). Lastly, the participants of fifth study composed of 91 students from the education faculty from a university in the Central Black Sea Region of Turkey. The age range of participants was from 18 to 25, and the average age was 19.97 (S.D.: 1.64). Most of the students were in their freshmen year (n=48, 53%), and had not sought any psychological help before (n=82, 90%). 2.2 Development Process of Barriers to Seeking Psychological Help Scale (BSPHS) Initially, the existing literature was reviewed so as to develop content domain for the BSPHS. Specifically, we reviewed previous studies that empirically examined inhibiting factors in psychological help seeking in student or adult samples (e.g., Calloway, 2008; Setiawan, 2006; Stanhope, 2002). Moreover, two open-ended questions were asked of students to determine encouraging and inhibiting factors in the help-seeking decision. These questions were Published by Sciedu Press 50 ISSN 1927-6044 E-ISSN 1927-6052 www.sciedupress.com/ijhe International Journal of Higher Education Vol. 6, No. 1; 2017 “What are the ‘facilitative/encouraging’ factors that help you receive psychological help?” and “What are the ‘inhibiting/hindering’ factors that stop you from receiving psychological help?” The students were asked to answers these questions based on their own opinions and previous experiences. Using students' answers to the questions as well as items we constructed based on the literature review, an item pool was generated including 35 items. After constructing the item pool, a draft scale was prepared accordingly and sent to seven experts. There were three experts in Guidance and Psychological Counseling, two experts in the Turkish Language and two experts in Measurement and Evaluation. After the definition of barriers related to psychological help seeking (Kuhl et al., 1997), experts rated each draft item’s clarity and content validity on a five-point scale ranging from Absolutely unsuitable (1) to Absolutely suitable (5). We calculated average expert ratings for each draft item based on clarity and content validity scores separately and then excluded them if the mean expert rating was below three for either the clarity or content validity score. As a result of this process, we removed 18 items from the draft scale, and finally, the 17-item BSPHS was constructed. 2.3 Other Scales In order to determine convergent and divergent validity estimates of BSPHS in fourth sample, we used The Self Concealment Scale, The Self-Stigma of Seeking Help Scale, and The Attitudes towards Seeking Psychological Help Scale-Revised. Psychometric properties of these scales is discussed in further detail below. Self Concealment. The Self Concealment Scale (SCS) was used to measure the self-concealment levels of college students. The SCS was developed by Larson and Chastain (1990) and adapted to Turkish culture by Terzi, Güngör, and Erdayı (2010).This one-dimensional scale measures people's tendencies to actively conceal personal information about secrets, negative and distressing life experiences, and thoughts about themselves. The SCS consists of 10 items and participants are asked to indicate to what extent they agree or disagree with each item on a five-point scale ranging from Strongly Disagree (1) to Strongly Agree (5). The scale does not include reverse coded items, and the scores of the scale range from 10 to 50. Higher scores indicate higher levels of self-concealment. The internal consistency coefficient of the scale for this study was (α) .78. An example item from the SCS is “I have an important secret that I haven’t shared with anyone.” Self-Stigma. The Self-Stigma of Seeking Help Scale (SSOSHS) is used to measure self-stigma associated with seeking psychological help. The SSOSHS was developed by Vogel, Wade, and Haake (2006) and adapted to the Turkish language by Topkaya (2011). Both the original and adapted scale is a one-dimensional structure consisting of 10 items and the participants respond to each item on a five-point Likert scale ranging from Strongly Disagree (1) to Strongly Agree (5). Items 2, 4, 5, 7, and 9 are reverse scored and then all items are summed for a total score. The total scores that can be obtained from the scale range from 10 to 50. However, the SSOSHS’s tenth item was not included in this study for statistical analysis because it exhibited very low item-total correlation. Thus, the scores of the SSOSHS range between 9 and 45 for this study. Higher scores indicate greater self-stigma associated with receiving psychological help from a mental health professional (Topkaya, 2011). The internal consistency coefficient (Cronbach’s Alpha) of the scale for this study was .70. An example item from the SSOSHS is “Seeking psychological help would make me feel less intelligent.” Attitudes towards Seeking Psychological Help. The Attitude towards Seeking Psychological Help Scale-Revised (ATSPHS-R) is used to measure people’s attitudes towards seeking psychological help. This scale was developed by Türküm (2001) using the previously validated Attitude towards Seeking Psychological Help Scale in the context of Turkish culture. The ATSPHS-R is comprised of 18 items and each item is rated on a five-point scale ranging from Strongly Disagree (1) to Strongly Agree (5). The total scores that can be obtained from the scale range from 18 to 90. High scores indicate more positive attitudes towards seeking psychological help. The test-retest reliability across the 10 weeks was measured at .77 and the Cronbach Alpha internal consistency of the scale in college students was .90 (Türküm, 2001). The Cronbach Alpha internal consistency coefficient of the scale for this study was .87. An example item from the ATSPHS-R is “When a close friend of mine asked me opinions about his/her mental health difficulties, I can recommend him/her to seek psychological help.” 2.4 Procedure Approval to conduct the study was obtained from Human Subjects Ethics Committee of related universities prior to study commencement. Following the ethics committee approvals, the participants were asked to complete the questionnaires, which were delivered to the participants by the first or third researcher at the beginning or end of their courses during regular class hours. Ethical considerations of the study clearly assured before administration of questionnaires. The students were informed about voluntary participation, confidentiality, anonymity, and that they could withdraw at any step of the study without any sanction. Students did not receive any incentives in return for Published by Sciedu Press 51 ISSN 1927-6044 E-ISSN 1927-6052 www.sciedupress.com/ijhe International Journal of Higher Education Vol. 6, No. 1; 2017 their participation in the study and all students participated voluntarily. All participants in the studies completed the BSPHS. The participants in the fourth sample also completed the SCS, SSOSHS, and ATSPHS-R. The BSPHS was administered to participants in the fifth sample again after three weeks for calculating test-retest reliability. Participants completed the questionnaires in about 20 to 30 minutes depending on the studied sample. 2.5 Statistical Analyses The statistical analyses were performed with SPSS 23©, Factor 10.3 (Lorenzo-Seva & Ferrando, 2006, 2013), and Mplus 7.4 (Muthén & Muthén, 1998-2014) statistical computer programs. Among all studies, 43 participants who did not answer most parts of the questionnaire were excluded from the data set. In the fifth sample, which was conducted to determine test-retest reliability, some of the participants could not answer all of the items. Specifically, there were five participants who could not answer only one item, and two participants who could not answer two items in the first administration. In the second administration, one participant did not answer three items, one participant did not answer two items, and three participants did not answer one item. For this limited number of participants with missing values, we replaced these missing values using the expectation-maximization missing data estimation method. Univariate and multivariate outliers were checked, and eight multivariate outliers in the second sample and six multivariate outliers in the third sample were excluded from the data set (Hair, Black, Babin, & Anderson, 2014). In order to determine underlying factor structure of the BSPHS, an exploratory factor analysis was conducted with the second study data. Before conducting the exploratory factor analysis, as suggested by measurement and evaluation experts (Costello & Osborne, 2005; Fabrigar, Wegener, MacCallum, & Strahan, 1999; Izquierdo, Olea, & Abad, 2014), several tests were performed in order to determine whether the correlation matrix of the data was suitable for exploratory factor analysis and to ascertain the number of factors to retain. Firstly, whether there was an adequate level of correlation among scale items was examined through Bartlett’s test of sphericity. Also, whether the sample size was sufficient to conduct factor analysis was analyzed with the Kaiser-Meyer-Olkin sampling adequacy coefficient. The Barlett Tests for Sphericity test whether the correlation matrix of the data is an identity matrix. To be able to conduct a factor analysis, the Barlett Tests for Sphericity results should be significant, and the Kaiser-Meyer-Olkin (KMO) sample adequacy coefficient should be .60 or above (Kaiser, 1974). In order to determine the number of factors to retain in the scale, multiple factor retention decision criteria, including eigenvalue greater than one, scree-test, parallel analysis, Velicer’s Map Test, and Hull method were used. Additionally, the coherence among scale items and the interpretability of the factors were other criteria. Also, another criteria was that there should be at least three items in each subscale (Hair et al., 2014). The eigenvalue greater than one rule, also known as in the literature as the Kaiser-Gutman Rule, posits that factors with an eigenvalue higher than one significantly contribute to the explained variance rate and the factors with an eigenvalue less than one contribute to explained variance less or trivial amount. Thus, the eigenvalue greater than one rule recommends to retain factors with eigenvalues higher than one. The Scree test aims to determine the number of factors to retain by visually analyzing eigenvalues in a scree-plot graphic. When the eigenvalues occur in a broken-stick pattern, researchers can find the true number of factors to retain. Parallel analysis produces random correlation matrixes with the same number of items and sample size in the scale and calculates the average eigenvalues using this virtually obtained correlation matrix, then the researchers compare these virtually obtained mean eigenvalues with their actual sample eigenvalues. The last sample eigenvalue higher than the virtually obtained eigenvalue determines the factor number in the scale. Velicer’s Map test states that the common factors in a scale are associated with latent factors, and these common factors show partial correlation with latent factors until the last true factor in the scale. In other words, Velicer’s Map test stops to extract the factors when partial correlation reaches the minimum level. Velicer’s Map test uses principal component analysis to calculate partial correlations. Lastly, the Hull method aims to find a model with the optimal balance between model and number of factors using goodness of fit indexes (e.g., Comparative fit index (CFI), Root Mean Square Error of Approximation (RMSEA)) (Cattell, 1966; Horn, 1965; Lorenzo-Seva & Ferrando, 2006; Lorenzo-Seva, Timmerman, & Kiers, 2011; Velicer, 1976). After determining the number of factors in the scale, exploratory factor analysis was performed with Principal Axis Factoring and Promax rotation (k=4). The reason for using Principal Axis Factoring is that this factoring approach does not require that scale items meet specific assumptions, such as multivariate normal distribution (Fabrigar et al., 1999). The reason for using oblique Promax rotation is that we expect correlations among obtained factors (Costello & Osborne, 2005) and there was no theoretical assumption demonstrating that the obtained factors would be independent from each other. At the same time, in order to determine which item belonged to which factor, we used item factor loading values of .30 or higher (Field, 2013), and there was to be a difference of at least .10 or higher between corresponding salient item factor loadings and other factors. Published by Sciedu Press 52 ISSN 1927-6044 E-ISSN 1927-6052 www.sciedupress.com/ijhe International Journal of Higher Education Vol. 6, No. 1; 2017 A confirmatory factor analysis was conducted with the third sample to cross-validate the underlying factor structure of the BSPHS in a new sample. The robust maximum likelihood method (MLR) was used as an estimation method in the confirmatory factor analysis. The MLR method corrects standard errors when data violate the multivariate normal distribution assumption. In confirmatory factor analysis, the hypothesized model’s fit to data is examined through a number of goodness of fit statistics, including adjusted Chi-Square (χ2 /df), RMSEA with 90% confidence interval, CFI, Tucker-Lewis Index (TLI), Standardized Root Mean Square Residual (SRMR) (Hu & Bentler, 1999; Marsh, Balla, & McDonald, 1988). Values less than 5 but more than 3 are acceptable for χ2 /df, and values less than 3 indicate a perfect fit. For RMSEA, the values between .08 and .06 show good fit, while values less than .06 indicate excellent fit. For CFI and TLI values, .90 and .95 indicate acceptable and excellent fit. Lastly, SRMR values of .08 and less indicate good fit (Hu & Bentler, 1999; Marsh et al., 1988). Lastly, in order to determine convergent and divergent validity, BSPHS subscales’ correlation with other theoretically relevant scales were examined with Pearson correlation analysis in the fourth sample. Similarly, the Pearson correlation analysis was also used with the fifth sample in order to determine the stability coefficient of BSPHS subscales, while Cronbach Alpha reliability was used to estimate the internal consistency of the subscales and total scale. 3. Results 3.1 Study 2: The Determination of Factor Number and Exploratory Factor Analysis Bartlett’s Test of Sphericity found that the correlation matrix was not an identity matrix (χ² =1417.30, p <.001), (136) and the Kaiser Meyer Olkin sampling adequacy coefficient was .86, well above the suggested minimum value for conducting a factor analysis (Kaiser, 1974). The item level sampling adequacy coefficient (anti image correlation) also ranged from .69 to .92. These findings suggested that the correlation matrix of the scale items was appropriate for conducting factor analysis. The eigenvalue greater than one rule found that the correlation matrix of the scale items consisted of five factors with an eigenvalue higher than unity. The scree plot results also showed a broken-stick pattern in factors two and five. While Horn’s (1965) parallel analysis suggested the extraction of the two factors, Velicer’s MAP test recommended the extraction of only one factor. Similarly, Hull’s method also suggested retaining one factor. Since multiple factor retention decision criteria suggested extracting different numbers of factors, we performed three separate factor analyses using one-, two-, and five-factor forced solutions with Principal Axis Factoring. When we extracted two or five factors, as previously specified the oblique Promax rotation (k=4) was used. It was found that forced one-factor and two-factor solutions did not reveal an interpretable factor structure. However, a forced five-factor solution yielded an interpretable factor structure and good coherence between items in each factor. Table 1 shows a forced five-factor solution using Principal Axis Factoring and Promax rotation. As seen in Table 1, Factor I consists of four items and factor loadings ranged from .74 to .94, which explains 34.24% of the total variance. When item contents were scrutinized, this factor indicated the people’s fears related to being stigmatized by society when they receive psychological help. Thus, this factor is named fear of being stigmatized by society. Factor II similarly includes four items with factor loadings of .31 to .81. This factor explains 10.30% of the total variance. Since this factor includes items related to the difficulties people feel trusting mental health professionals, this factor is called trust in the mental health professional. Factor III consists of three items with factor loadings that ranged from .64 to .87. This dimension reflects the items related to people’s self-disclosure to a mental health professional, thus it was called difficulties in self-disclosure. Factor IV includes three items with factor loadings of .40 to .86. This factor consists of items that evaluate people’s negative thoughts about themselves when they receive psychological help. Thus, this factor is called perceived devaluation. And lastly, Factor V includes three items with factor loadings ranging from .33 to .95. This factor contains the items regarding people’s lack of knowledge about seeking psychological help, so it is called lack of knowledge. The BSPHS is presented in Appendix A. 3.2 Study 3: Confirmatory Factor Analysis A confirmatory factor analysis was conducted to test whether the BSPHS would have a similar factor structure in a new sample. Based on current factor analysis and different factor retention decision criteria results, three possible competing models were tested in the confirmatory factor analysis in the third study: One-factor model (Model 1), two correlated factor model (Model 2), and suggested five correlated factor model (Model 3). The two-factor model estimated was based on exploratory factor analysis results. Table 2 shows the goodness of fit indexes of competing models based on confirmatory factor analysis. Published by Sciedu Press 53 ISSN 1927-6044 E-ISSN 1927-6052 www.sciedupress.com/ijhe International Journal of Higher Education Vol. 6, No. 1; 2017 Table 1. Exploratory Factor Analysis Results of BSPHS Factor Loadings Factors I II III IV V h2 Item9 .94 .81 Item2 .83 .67 Item6 .81 .77 Item14 .74 .64 Item16 .81 .63 Item17 .80 .60 Item12 .50 .35 Item4 .31 .29 Item5 .87 .73 Item8 .70 .52 Item1 .64 .50 Item13 .86 .77 Item11 .68 .49 Item15 .40 .31 Item10 .95 .88 Item7 .44 .21 Item 3 .33 .19 Eigenvalue 5.82 1.74 1.42 1.20 1.14 Explained Variance (%) 34.24 10.30 8.33 7.08 6.71 Note. h2 = communality. Absolute factor loading values less than .30 were discarded for the sake of clarity. Table 2. Goodness of Fit Indexes of Competing Models Model χ2 df χ2 /df CFI TLI RMSEA 90 % CI SRMR One Factor 408.026 119 3.429 .719 .679 .105 .094-.116 .085 Two BSPHS Factors 368.697 117 3.145 .756 .716 .098 .087-.110 .087 Five BSPHS Factors 138.343 109 1.269 .972 .964 .035 .012-.051 .053 Note. The two-factor model estimated is based on exploratory factor analysis results. Correlations between latent factors in two-factor model and five-factors models were allowed. Latent factor variances were fixed to one for model identification purposes. C.I.: Confidence Interval. As seen in Table 2, the one-factor model and two-factor model showed poor fit to data. However, the five correlated factor model showed excellent fit to data. These findings confirmed the factor structure of the scale which was obtained from exploratory factor analysis in a new sample. Table 3 demonstrates the standardized item factor loadings, t-values, and squared multiple correlations. As seen in Table 3, all factor loadings and t-values were large and statistically significant. Published by Sciedu Press 54 ISSN 1927-6044 E-ISSN 1927-6052 www.sciedupress.com/ijhe International Journal of Higher Education Vol. 6, No. 1; 2017 Table 3. Results of Confirmatory Factor Analysis Standardized Factor Loadings I II III IV V t R2 Factors Item9 .81(.05) 16.64 .66 Item2 .90(.03) 36.51 .81 Item6 .91(.02) 39.38 .82 Item14 .71(.06) 12.29 .51 Item16 .77(.05) 15.12 .60 Item17 .72(.05) 14.16 .52 Item12 .72(.06) 12.50 .51 Item4 .49(.06) 8.34 .24 Item5 .81(.05) 18.17 .66 Item8 .77(.04) 17.45 .59 Item1 .66(.07) 10.00 .43 Item13 .81(.05) 16.73 .65 Item11 .79(.08) 10.07 .62 Item15 .57(.07) 8.67 .33 Item10 .78(.07) 11.89 .60 Item7 .75(.05) 13.98 .56 Item 3 .68(.07) 9.49 .47 Note. Numbers in parenthesis are standard errors of factor loadings. All t-values and R2 values were significant at p<.001. 3.4 Study 4: Convergent and Divergent Validity In order to determine convergent and divergent validity of the BSPHS, the relationship between the subscale scores of the BSPHS and attitudes towards seeking psychological help, self-concealment, and self-stigma scores were examined using Pearson correlation analysis. It was expected that, as an individual’s barriers to seeking psychological help increased, attitudes towards seeking psychological help scores would decrease and self-concealment and self-stigma scores would increase. Table 4 presents the correlations among the study variables. As shown in Table 4, the subscale scores of the BSPHS are positively correlated with each other in the small to medium range. Moreover, the subscale scores of the BSPHS are positively correlated with self-stigma and self-concealment scores, evidencing convergent validity of the BSPHS, whereas the subscale scores of the BSPHS are negatively correlated with attitudes towards psychological help-seeking scores, evidencing divergent validity of the BSPHS scores. In other words, as individuals’ barriers related to fear of being stigmatized by society, trust in the mental health professional, difficulties in self-disclosure, perceived devaluation, and lack of knowledge increase, the self-concealment and self-stigma levels tend to increase and they tend to hold more negative attitudes towards seeking psychological help. These findings support convergent and divergent validity of the BSPHS. Published by Sciedu Press 55 ISSN 1927-6044 E-ISSN 1927-6052 www.sciedupress.com/ijhe International Journal of Higher Education Vol. 6, No. 1; 2017 Table 4. Correlations among Variables 1 2 3 4 5 6 7 8 1. Fear of being stigmatized by society 2. Trust in the mental health professional .43 3. Difficulties in self-disclosure .52 .44 4. Perceived devaluation .52 .50 .51 5. Lack of knowledge .41 .37 .36 .29 6. Self-stigma .29 .32 .22 .24 .22 7. Self-concealment .28 .24 .25 .21 .13* .33 8. Attitude towards seekingpsychological help -.23 -.31 -.22 -.28 -.28 -.61 -.27 Mean 7.71 10.07 7.77 6.52 6.57 21.74 24.57 65.92 S.D. 3.30 3.15 2.69 2.44 2.41 6.07 7.19 11.51 Note. p<.05*. All other correlations were significant at p<.01. 3.5 Study 2, 3, and 5: Reliability of the BSPHS The reliability of the BSPHS was examined through test-retest and Cronbach Alpha internal consistency coefficient. The test-retest reliability study results obtained though the administration of the scale twice within a three-week interval produced.66 for fear of being stigmatized by society, .69 for the trust in the mental health professional, .71 for difficulties in self-disclosure, .56 for perceived devaluation, and .66 for the lack of knowledge subscales. Table 5 demonstrates the Cronbach Alpha internal consistency coefficients obtained from the second and third samples. Table 5. Cronbach Alpha Internal Consistency Coefficients across Different Samples Study 2 Study 3 Scale α α Fear of being stigmatized by society .91 .90 Trust in the mental health professional .73 .77 Difficulties in self-disclosure .77 .78 Perceived devaluation .71 .75 Lack of knowledge .58 .78 Total scale .87 .91 As seen in Table 5, the Cronbach Alpha internal consistency coefficient of the scale in the second sample ranged from .58 (lack of knowledge) to .91 (fear of being stigmatized by society), while the Cronbach Alpha internal consistency coefficient of the scale in the third sample ranged from .75 (perceived devaluation) to .90 (fear of being stigmatized by society). The Cronbach Alpha internal consistency coefficient of the total scale was .87 for the second sample and .91 for the third sample. 4. Discussion The college years is a period when most students have to cope with new challenges and new experiences. They may face some personal, social, and academic problems, which may require them to seek psychological help in this period (Güneri, Aydın, & Skovholt, 2003; Koydemir et al., 2010; Setiawan, 2006). However, studies have consistently shown that their willingness to seek psychological help to cope with these problems is often not in accordance with the intensity of their problems (Mansfield et al., 2005; Setiawan, 2006). Moreover, it is obvious that there are various factors influencing whether university students seek psychological help from a mental health professional (Kakhnovets, 2011; Mansfield et al., 2005; Setiawan, 2006). Unfortunately, there is no instrument to determine these factors and how they influence individuals seeking psychological help, or which factors are more important than others in this process. Thus, the purpose of this study was to develop an instrument to evaluate the barriers affecting university students seeking psychological help. Consistent with this purpose, firstly, the possible factors influencing the intentions of university students to seek Published by Sciedu Press 56 ISSN 1927-6044 E-ISSN 1927-6052 www.sciedupress.com/ijhe International Journal of Higher Education Vol. 6, No. 1; 2017 psychological help were determined through an extensive literature review and the use of two open-ended questions to uncover facilitating and inhibiting factors. After the content analysis and literature review, a 35-item draft form was prepared. It was then presented to experts for review. Based the results, 18 items were discarded from the draft and the final form of the scale was created with 17 items. Exploratory factor analysis results suggested that the scale should consist of five dimensions: fear of being stigmatized by society, trust in the mental health professional, difficulties in self-disclosure, perceived devaluation, and lack of knowledge. These dimensions explained 66.65% of the total variance. The explained total variance rate in the literature for multidimensional instruments accounting for 50% or more is accepted as adequate for scales (Beavers et al., 2013). In this respect, the BSPHS is an adequate scale to measure factors that inhibit college students from seeking psychological help. In the confirmatory factor analysis with a different sample, three competing models were tested. The results suggested that the goodness of fit index of one-factor and two-factor models did not fit the sample data, but the goodness of fit index of the five-factor model was perfect. Thus, the factor structure obtained from the exploratory factor analysis was confirmed in a new sample. The results of this study lend further support to previous studies showing that stigma, trust and confidentiality, lack of knowledge, difficulties to express emotions is a barrier in psychological help-seeking process (Kakhnovets, 2011; Kuhl et al., 1997; Setiawan, 2006). The convergent and divergent validity estimates of the scale were examined by correlating relevant psychological constructs, including attitudes towards seeking psychological help, self-concealment, and self-stigma associated with receiving psychological help. Previous research has shown that there was a negative relationship between self-stigma and attitudes towards seeking psychological help, between public stigma and attitude towards seeking psychological help, and between self-concealment and attitudes towards seeking psychological help (Li, Dorstyn, & Denson, 2014; Nam et al., 2013). Thus, it was expected that there would be a positive relationship between barriers to seeking psychological help subscales and self-stigma and self-concealment, while there would be a negative relationship between the attitudes towards seeking psychological help. In the study, it was observed that the subscales of the BSPHS were weakly but positively correlated with self-stigma and self-concealment. Moreover, the subscales of the BSPHS were also weakly but negatively associated with the attitudes towards seeking psychological help. These findings support convergent and divergent validity of the BSPHS. Lastly, the reliability of the scale was examined using test-retest reliability and Cronbach Alpha internal consistency coefficient. Reliability of the BSPHS was assessed via test-retest correlations using a three-week time interval that revealed that the stability coefficient was in the acceptable to good range(between .56 and .71). Additionally, although the subscales of the BSPHS consisted of a limited number of items, the internal consistency coefficients obtained from the different samples were mostly above .70. According to Nunnally and Bernstein (1994), a threshold value of reliability of .70 or above for research and screening purposes and .90 for clinical practice is considered adequate. In this sense, the BSPHS may be used as an appropriate scale for research and screening purposes in college students. The BSPHS is a self-reported scale, in which the participants specify their levels of agreement with each item by marking one of the options ranging from 1 (Strongly Disagree) to 5 (Strongly Agree). There is no reverse coded item in the scale. The total scores that can be obtained from the fear of being stigmatized by society and trust in the mental health professional subscales range from 4 to 20. The scores that can be obtained from difficulties in self-disclosure, perceived devaluation, and lack of knowledge subscales range from 3 to 15. Higher scores indicate higher obstacles in the related subscale and higher perceived obstacles in the related scale. In comparing the dimensions, the score an individual obtains from the related subscale is divided into the item number in that dimension, thus scores ranging from 1 to 5 may be obtained. Then, by comparing the means of these scores, it can be determined in which dimension an individual perceived more or fewer barriers. This study has some limitations that could provide opportunities to conduct new psychometric studies of the BSPHS. Firstly, the validity and reliability studies were conducted with mostly non-clinical Turkish university students from two public universities located in urban areas of Turkey. Thus, external validity of our study is low. Future studies could examine the validity and reliability of the scale on various samples, including adults and clinical samples, as well as examining cross-cultural validity and reliability. Secondly, although factorial, convergent, and divergent validity studies of the scale were conducted, the predictive validity of the scale has not been examined. Future studies could also investigate to what extent the scale predicts actual psychological help seeking behavior in college students. In conclusion, considering the psychometric properties of the scale, we found initial support for the validity and reliability of the BSPHS for determining the barriers seeking psychological help in university students. Determining these barriers may help contribute to prevention and intervention efforts aimed at promoting mental health service Published by Sciedu Press 57 ISSN 1927-6044 E-ISSN 1927-6052