Creativity of images: using digital consensual assessment to evaluate mood boards

Defining creativity

As can be inferred from these various definitions, there are multiple levels of creativity definition, with a few recurring thematic concepts. Generally, a person or product is considered to be creative based on novelty to the domain or field of work. In addition to novelty, the creative process and output evaluation and acceptance are commonplace in creativity definitions. Since the early 1950s research in the area of creativity has led to acceptance of the following four areas of focus (Four P’s) in creativity studies: (1) person, (2) process, (3) product and (4) press (environment) (Kaufman and Baer 2012). Initially, much of the research completed focused on the person and personality, but contemporary theoretical models exhibited the influence of the other three major concepts and show the relevance of evaluation and assessment of the creative product in relation to fashion designers and the fashion design process.

Fashion mood boards

A powerful communication tool, the fashion mood board comprises incongruent and seemingly un-related images which when tactfully selected provide complementary support in creating a coherent visual message discussing a specified theme or idea (Boyes 1998). Yet, the currently available research specific to mood boards is lacking and limited to a few studies within the past few years. Mood boards are a tool used to visually communicate information in fashion-related and consumer product industries (Cassidy 2011). Primary uses include bringing together images and ideas into an aesthetically and creatively impactful workspace, for a specific purpose. From an academic standpoint the inclusion of mood boards on projects and within courses provides training for industry practice, while providing students with the opportunity to communicate their ideas visually. Within this space, designers select and arrange images, artifacts, colors, fabrics, etc. in a calculated and planned effort to link content to a particular theme or idea. However, there exists limited training or skill development related to mood boards in academia, especially in relation to mood board evaluation and improvement, as well as the role in their role in the design process (Cassidy 2011).

Categorically, Cassidy (2008) defined four types of mood boards and their specific uses as indicated following. The mood boards used are examples created within the past 5 years by students for a course project. While these boards may not score high on creativity assessment, boards were purposely selected as evidence specific to the various categories identified by Cassidy (2011).

Category 1 boards (Fig. 1) are focused on target market identification, including lifestyles, socio-cultural factors, demographics, personal and cultural values/norms, and links between company and target market values. These boards are often labeled as lifestyle, target, customer, or consumer profile boards.

Category 2 (Fig. 2) is more conceptual in comparison and utilized during the initial stages of the product design process. Often focused on a feeling or exploration of ideas, board content is typically non-specific and derived from target market lifestyles. Through visual communication, these boards provide clarity and vision for a thematic concept derived from design briefs. Most often these are referred to as concept, idea, inspiration, story or style boards. These boards are conceptual in content and interpretation and therefore will be the focus of this investigation.

Category 3 boards (Fig. 3) enhance and sharpen the concepts and ideas developed in category 2 boards. Specificity is more apparent at this level and includes a variety of boards with clearly defined purpose: color, fabric, style, trend, samples and forecasting. Much of the work utilizing boards at this stage are focused on refining the product identity and message, through visual representation and comparison.

Category 4 boards (Fig. 4) are the most refined and professional boards, as they represent the final product line carried to clients and consumers. Often used by marketing and merchandising teams, the presentation or usage boards in this stage connect the product with the overall brand image and identity.

Mood boards encompass various aspects of the fashion industry and contribute significantly at each stage of the product development life cycle. Additionally, mood boards enhance creative thinking and fluency of ideas as well as exploration about the mode and content of the products’ visual message. Designers’ creative expression and experimentation during the mood board creation allows them to dive deeper into themes or concepts, providing a wealth of information for designers to pull inspiration from. Lastly, through the use of mood boards, designers may be able to communicate visually and effectively, what is a web of seemingly unconnected ideas, difficult to express verbally with similar impacts. In essence, the mood board serves to engage designers in deeper creative thinking, yet evidence is lacking regarding a formalized and reliable assessment instrument to provide critical feedback for mood board creation skill development in academia.

Consensual assessment technique

According to creativity researchers based in psychology and cognitive sciences, there are multiple levels of creativity, with a few recurring thematic ideas (Amabile 1996; Gardner 1993; Runco 2007; Simonton 2009). Generally, a person or product is considered to be creative based on novelty and usefulness to the domain or field of work. In addition to novelty, the creative process and output evaluation and acceptance are commonplace in creativity definitions (Amabile 1983a, 1996; Gardner 1983, 1993; Guilford 1950; Sawyer 2006; Torrance 1962). Early work conducted by Amabile (1983a, b, 1996) developing the Consensual Assessment Technique (CAT) is the foundation for numerous research studies and creativity theory development across domains. The original CAT contained sixteen items evaluating creativity and technical quality. The CAT is well validated in research studies and provides a reliable creativity assessment instrument using a range (expert to novice) of raters (Amabile 1983a, b, 1996; Baer 1997, 1998; Baer et al. 2004; Kaufman et al. 2005, 2007, 2008; Runco 1999). Studies conducted across domains using both adults and children as subjects, have yielded reliability results often exceeding .70 (see Amabile 1996). With early editions, a panel of expert raters independently evaluated parallel products on numerous measures of creativity and technical quality.

Condensed versions (single-item measures) of the CAT have offered similar reliability of creativity assessment, while facilitating the collection of assessment data. For instance, Kaufman et al. (2008) evaluated SciFaiku (Japanese variation on the haiku) poetry written by volunteers. In this study, both novice and expert raters conducted an assessment using a single-item measure asking them to rate the creativity of the poem on a scale of 1–6. Overall, reliability analyses for the separate groups were reported as good (>.80) and excellent (>.90). In a related and similar experiment, Kaufman et al. (2009) used a single question evaluation for creativity to assess short stories, with similar results. Expert and novice film ratings were evaluated using a single-item measure (rating scale 1–10) with success using students, online ratings, and expert film critics (Plucker et al. 2009). Successful results using the CAT are often defined as high levels of interrater reliability (>.80) within and between groups of expert and non-expert raters. This definition is used for the overall assessment of the CAT in the current study. In various domains, researchers using single-item measures of creativity have reported success using a condensed CAT, while reporting no administration issues, questions of reliability, nor validity concerns.

Expert vs. non-expert raters

In certain domains, would non-experts provide an equally accurate assessment of creativity than experts? While the cost and time of expert panels presents difficulties in creativity assessment, the use of non-experts may not provide a valid assessment of creativity, despite reliability measures (Kaufman and Baer 2012). Early research into the use of experts vs. non-experts indicates the incongruence in responses and creativity assessments; yet with sufficient interrater reliability within groups.

Kaufman et al. (2008) conducted a study involving experts and non-experts in a specified technical domain (poetry) and reported satisfactory to high interrater reliabilities (expert: α = .83; non-expert: α = .94). However, sample sizes of raters (expert: n = 10; non-expert: n = 106) effected interrater reliabilities of non-experts; when adjusted for size, non-expert interrater reliabilities dropped significantly to α = .58. With little correlation between the groups, conclusions do not support the use or substitution of non-experts in this particular poetry domain. In a less specialized and familiar domain (short-story writing), results using high numbers of non-experts (n = 100+) correlated significantly with those of experts (r = .71), yet when interrater reliabilities are adjusted for sample size non-expert ratings dropped significantly to α = .53 (Kaufman et al. 2009). In repeated studies, using both experts and novices; children and adults; verbal and visual products the CAT provided satisfactorily interrater reliabilities, which usually exceed .70, with some reported as high as the mid .90s. In a majority of these studies, interrater reliabilities typically ranged in the .80s (Amabile 1983a, b, 1996; Baer 1997, 1998; Baer et al. 2004; Kaufman et al. 2005, 2007, 2008; Runco 1999).

In early studies using the CAT to assess creativity levels in collages and drawings of children and adults, Amabile (1983a, 1996) reported inter-rater reliabilities from .70–.90. In subsequent studies focused on image and story creation in children, similar inter-rater reliabilities were reported (Hennessey and Amabile 1999; Runco 1999). In subsequent studies, Kaufman et al. (2005), found gifted novices provided reliable ratings, which were highly correlated with those of expert judges. Plucker et al. (2008) investigated the use of expert, quasi-expert (via professional and user-driven websites), and novice ratings of more than 600 movies. While novice and expert correlations were similar to earlier investigations cited on novice vs. expert ratings, results of the quasi-expert group bridged the gap with significant correlations with both novice (r = .65) and experts (r = .72). Results indicate a dichotomous classification of raters does not support the full range of options available to researchers for creativity evaluation (Plucker et al. 2008, 2009). Therefore the use of gifted novices or quasi-experts is supported in conjunction with, or as a suitable substitute for, expert ratings of creative products in technical and non-technical domains (Baer et al. 2004, 2009; Kaufman et al. 2005).

Domain specificity of evaluation

Creativity varies across domains, as does the expertise required for valid assessment using the CAT (Baer 1993, 1996). In previously cited studies, evaluation comparisons between experts, quasi-experts and novices indicated varied results within the domains of poetry, film, writing, etc. While examining the creativity of captions, Kaufman et al. (2007) indicated psychology students maintained adequate interrater reliability and sufficient ratings across writers. Therefore, the use of students evaluating written captions was acceptable.

In an area with no discernable training or apprenticeship, such as mood boards or image captions, can an amateur or a quasi-expert from a related field provide a reliable and valid assessment of creativity? Kaufman and Baer (2012) suggest the use of novice raters for expert-less domains may provide reliable creativity assessment, yet questions of validity would remain of concern. While results from creativity assessment in these domains will produce some validity, further support may be given by comparison between novice and quasi-experts from a related field. Additionally, the use of those involved in the creation or the projects as self and peer assessors may provide background knowledge of the project and increase validity of the creativity assessment. Based on an inversion of the hierarchy of domains (Simonton 2009), the potential or assessment of personal and/or everyday creativity is supported by the through the originality of the idea, even if only for the individual. Based on this premise, in assessing creativity in an expert domain, the use of participant artists/raters seems appropriate with sufficient validity.

Questions linger about the domain specificity of creativity assessment and methods of identifying and classifying experts. Plucker and Runco (1998) stated the uselessness of single predictive creativity measures (e.g. creativity tests), and agreed with Hennessey and Amabile (1988) of using an overall assessment of product output, measured using consensual assessment. In prior research on expert and non-expert creativity assessment, the domains were limited to those ranking higher on Simonton’s (2009) hierarchy of domains, such as poetry, and creative writing. Fashion, by its very purpose and use of materials, has hinged between utilitarian artifact with functional appeal to creative applied arts, with exhibitions in the finest museums around the world. With its multiple functions, there exists a deficit in understanding and evaluating creative outputs, starting with the concept mood board stage. In addition, much of the prior research using verbal stimuli resulted in effective creativity evaluation, however assessment of figural stimuli is limited when comparing expert and non-expert assessments of creativity.

Procedure for mood board assignment

The five concept words were selected by the authors based on the vagueness of the term and openness to interpretation by a diverse group of participants. In reviewing a list of possible terms identified through the creativity literature, the researchers narrowed down to the five terms based on fit of the terms to the consensual assessment methodology (Amabile 1983a; Kaufman et al. 2005, 2007). The heuristic nature of the terminology and approach of the instructions were intentional as evidenced by prior creativity researchers when setting task parameters (Amabile 1983a, b, 1996; Baer 1997, 1998; Baer et al. 2004; Kaufman et al. 2005, 2007, 2008; Runco 1999). Photos, selected by the participants to represent their chosen theme, were uploaded onto a single page PDF and formatted to standardize size consistency, with the selected theme as the title heading. Instructors from two different universities asked students in their courses: creative thinking and design and visual design to complete the assignment, resulting in 50 total mood boards. Categorical themes were not even across groups, with students selecting and producing the following mood board themes: clever (n = 3), loud (n = 14), think (n = 5), religious (n = 16), and tired (n = 12). Figure 5 provides an example of the category 2 style concept boards evaluated by the raters.

Procedure for expert and non-expert raters

After all 50 students completed their mood boards; each file was formatted to maintain consistency for rating and evaluation. Students participated in the project, following IRB consent, based on enrollment in select courses. One of the courses used was specific to an overview of the fashion industry and is open to all majors. The other course specifically engaged in the concept of creativity as a key overarching theme in the fashion industry, again open to all majors. Both course are open to all classification levels, however most of the students enrolled are a second or third year level. Over 90% of the enrolled students are majoring in fashion design and/or fashion merchandising. The remaining 10% were from various majors in the visual arts. Images were transferred to an electronic survey, where raters were asked to rate the creativity of each collage of images in relation to the theme noted in text. A 5-point Likert scale was used representing the level of creativity for each collage. Ratings ranged form not creative (1) to very creative (5). Images were randomized for each rater, with raters evaluating all the mood boards. Expert raters (n = 10) were selected based on prior academic teaching in the fields of apparel design, visual arts, interior design, and/or architecture. Expert raters (n = 10) were purposively sampled and exhibited proficiency in their domain with creative works and scholarship accepted for juried and peer-evaluated national/international competitions. Expert raters were identified as having a knowledge in the area of visual representation of conceptual ideas, specifically fashion design mood boards. 8 of the 10 experts has an educational and/or occupational background in fashion design, while the other two expert raters’ educational and/or occupational background is in visual arts. However, both of the non-fashion design experts had worked (occupationally) in textile/fashion design at a stage in their career and exhibited a knowledge of the purpose and intent of conceptual mood boards. Non-expert raters (n = 88) included students who completed the projects as well as additional students studying fashion design and/or fashion merchandising across two universities. All 50 students who participated in creating a mood board also rated a mood board, however they did not rate their own mood board. The underlying theory of consensual assessment by non-experts indicates if a significant number of evaluators (>50) are used then individual rater bias is controlled through statistical power (Amabile 1996). Therefore, non-expert rater bias was not a concern for the scope of this project. A minimal amount of extra credit, representing <1% of the total course grade, was provided to students who completed the evaluation and rating.

Data analysis

During initial data analysis and correction, missing ratings were handled using techniques sufficient for data correction. Any rater who did not rate at least 75% of the images was omitted from further data analysis. Of the remaining, none were missing individual ratings for either expert or non-expert raters. Of the 105 non-expert raters who accessed the survey, 101 agreed to continue with the project. A total of 88 of the 101 completed the survey. Of the ten experts to access the survey all ten agreed to participate and fully completed the survey. Consistency among the raters was evaluated using a Cronbach’s coefficient alpha. As in many creativity studies coefficient alpha is calculated using the raters as variable items. If there is a difference in the number of raters for experts and non-experts, as is often the case, then an adjusted coefficient alpha should be used for interrater reliability comparison (Kaufman et al. 2008). Due to the size difference between the two groups of raters, n = 88 compared to n = 10, a Spearman-Brown adjusted coefficient alpha was used to standardize the internal consistency measures for both groups. In a comparison between coefficient alphas of expert and non-expert raters, confidence intervals were calculated using a standard error. Alpha level of .05 was set for the difference. An independent samples t test was calculated to compare means between the two groups for significant differences. Confident that appropriate assumptions for correlational analysis were met, a Pearson correlation between expert and non-expert ratings was conducted to assess the relationship between two group ratings.

A repeated measure of analysis (ANOVA) was computed to examine the differences among the creativity ratings’ means of both non-expert and expert raters and the subject matter of each of the mood boards (clever, loud, think, religious, and tired). Data analysis included examination of significant differences between subject matter categories for each group (non-expert and expert) as well as an overall assessment for all raters.

Implications

As creativity assessment moves across domains, this question regarding qualifications of judges will provide additional questions about the reliability and validity of consensual assessment. Prior research using consensual assessment, indicated substituting non-experts for experts was not recommended, and this investigation continues to support that position. Although, ideally combining the use of expert and non-expert raters, provides reliable assessments and validity support. Maintaining equal group sizes is recommended. Not only will students be provided with reliable feedback on creativity of content using this measure, but they will receive assessment from a variety of sources and backgrounds. By seeking more substantive feedback, students can focus mood board creation skill development on specific deficiencies. However, the first step in development and progress is a true and reliable assessment of current conditions. The ease of use of the single digital measure to evaluate creativity of visual stimuli provides academics and students an initial assessment of the quality of content selected prior to subjective aesthetic judgment.

Objectivity is becoming the norm in the current education grading systems. However, in disciplines such as apparel design or the applied arts, objectivity is often difficult to achieve and may lead the student to discredit creativity evaluation as one person’s opinion. Therefore, through the use of digital consensual assessment, the subjective expert opinion of a faculty expert is supported by the assessment of non-experts with an interest in the field (peers). This procedure will enable effective multi-channel feedback for the designer/artist and provide unique learning opportunities in receiving and reacting to critical assessments. While this tool will not fully replace the keen expert eye of a trained educator, use of this instrument will go a long way in helping to support the subjective assessment of creativity in academia.

Limitations of this study include unequal distribution of mood boards under each of the five categories. However, the effectiveness of using a single item digital measure for creativity evaluation is not diminished when compared with previous studies using consensual assessment. Additionally, the difference in group size between expert and non-expert raters is a consistent issue from the literature. While it would be ideal from a statistical analysis perspective to maintain equal groups, the difficulty in amassing a panel of experts is often a hindrance. Further examination to measure the reliability and validity of the instrument with equal sizes is needed and recommended for future studies. From an education and research perspective, the use of this instrument can provide an efficient and simple creativity evaluation measure with adequate reliability, continuing to promote the research and teaching of creativity with quantitative evaluation. Further studies examining variation in mood board styles and features as well as depth of assessment will provide additional scholarship contributing to the field. Additionally, identifying mood board creation skills will enable assessment research to target and evaluate specific skills for further enhancement. The use of mood boards to express the creativity of inspiration digitally will continue to grow in the fashion industry. Therefore, continued research evaluating, critiquing, and enhancing digital mood boards and creative expression will be necessary.