Thinking Like a Museum Evaluator

Poster Session conversations at the VSA Conference

In my next life I want to be a museum evaluator and researcher. I guess that will have to be after I am a museum planner, a librarian, an urban planner, and an arborist. I was reminded about how much I enjoy the evaluator perspective at the recent Visitor Studies Association conference in Chicago, July 24-27.

One thing I tuned into over the two-and-a-half day conference was how presenters, evaluators, and researchers answered questions. When a question was posed, I often noticed there was a short but perceptible pause, followed by a well-crafted reply. In the cartoon part of my brain, I pictured a thought bubble above her head with a list or steps in a logic model she checked before answering. I also thought I saw another thought bubble hovering over the head of someone whose answer noted specialized terms.

This is the second year I have attended the VSA conference. Both years I have been drawn to and invigorated by the thoughtful and disciplined thinking and interest in critical reflection in so many sessions and among so many people. Again and again, I saw a willingness, even a zest, to explore questions. I often had a sense of an evaluator’s independence, objectivity, and neutral stance related to information that was gathered, presented, and interpreted.

Thinking About Practice, Finding Patterns
Listening to projects, studies, methods, and results, I thought about the related thinking and practices that support this work. These practices not only made the sessions better, but are relevant for me in my work and in work across museum. Four sets of practices have stayed with me. 

Questions Everywhere. There’s no doubt about it. Evaluators and researchers love questions. Sessions opened with questions as titles and ended with questions as well. “Based on this, how can we create programs that…?” was a question I heard more than once. It wasn’t just the sheer quantity of questions that was impressive. They were well-crafted questions. Some framed major conversations, others took into account the complexity of situations: “How do we evaluate fairly when pupils have different social, economic, and family circumstances…?” Even standing at the elevator, in the poster session, or at lunch, evaluators and researchers were asking questions.

Checking and Challenging Assumptions. Checking assumptions is sometimes procedural like asking if everyone has had a chance to speak. Other times it seriously challenges the fundamentals like asking how (in)adequate attendance is as a measure of success. Checking assumptions can also flip a switch and reroute thinking, as when Joe Heimlich said that measures are met when we’re successful; we’re not successful because we meet our measures. There were many friendly provocations such as asking whether we are preserving success measures to preserve ourselves. Besides a willingness to challenge assumptions, I admired an appetite for experimenting in what presenters shared and encouraged others to try. In the middle of a session, one of the presenters reminded participants, “We’re trying to do new things.” 

Finding Language. I appreciate precise language for clarity, variety, and getting at meaningful distinctions. Multiple references to the role of language in visitor studies were made. In the first session I heard “languaging visitors,” or giving them the tools to talk about and share art. The value of visitors’ language to get at what’s intangible, like intrinsic benefits, also came up. For that matter evaluator and researcher language and “articulating intended results” received attention. There were many acknowledgements of context-specific terms: Big I and little “i” identity, inquiry-based strategies, and the Exploratorium’s own definition of “immersives.”

Playing It Forward. Frequently, the follow-up to presenting a project or a study was a slide or the question, “What can we do better?” If this question wasn’t posed, then a focus on Next Steps or areas of future study was. Often specific ways in which programs, exhibits, or marketing materials could be changed were highlighted. I greatly appreciated the push for improvement this represents­–a strong interest in action, change, and closer alignment of intention and achievement. Someone in the final session asked, “How do we, as a field, increase the rigor of our work in ways that are supportive of our colleagues but hard on the research and evaluation?”

Now, there’s a great question that challenges assumptions, makes meaningful distinctions, and plays the conference forward.

An Evaluator’s Perspective

New perspectives arriving

These practices interested me and I wanted to learn more about how evaluators see these and their own practices. During the conference I asked five people about what they see as a distinct perspective of an evaluator. We talked in line for the bus, at the elevators, and before sessions started. Speaking primarily about the perspective they bring to evaluation, they mentioned the following.

•                  I’m constantly wondering how the visitor experience will be. I look at prototyping as the value of the visitor’s report and behavior. (Elizabeth, in-house exhibit planner with evaluator responsibilities)

•                  Testing assumptions about what the visitor, or learner, takes away or understands from the experience. I try and bring multiple stakeholders and their perspectives to the task of interpreting what people are taking away. (Camilla, independent evaluator)

•                  I’m constantly asking questions and wanting to know the reason for things. Everything you figure out leads to a new question. (Lorrie, independent evaluator)

•                  I’m hungry for context. I ask myself if I have and understand the context I need (background, perspective, familiarity, etc.) to bring the right tools to this person or team to do what they need to accomplish. (Nina, in-house evaluator)

•                  Objectivity. An evaluator has a neutral relationship with information; she needs to show the information and let program people bring their perspective and needs to interpreting it. (Andréa, in-house evaluator)

Over the course of my five conversations, I shifted my question from asking about an evaluator’s thinking to an evaluator’s perspective. I debated about whether and when to tell them I was going to put their response in this blog and decided to do so. I did notice I had a mix of internal and external evaluators and researchers, but also realized by sample included only women. I wished I had more time for conversations and a new round of questions.

The conference was valuable as was my little study. I have a lot to play forward and a lot of work ahead before I’m thinking like an evaluator.

Learning Assets - Part 2

Lookout Cove: 2.5 acres of outdoor exploration at Bay Area Discovery Museum

In exploring museums' learning assets recently, I mentioned about 15 different substantial resources through which museums deliver experience and learning value to their visitors and communities. The list continues to grow through conversations and reading. But neither the precise list nor the number of items is really important. Instead it illustrates that museums have many learning assets. As you can see, most of them are quite familiar.

•           Exhibits, programs, collections, library, school or preschool, maker space, film and multi-media, historic structure, gardens, science park, the building, research center, outreach or mobile museum, recycle center, nature center, planetarium, aquarium, teacher resource center, theater, and hall of fame.

Every museum has learning assets. But not every museum knows what its learning assets are and why they are valuable. Museums do consider exhibits (or programs, teacher resource center, or aquarium) as financial assets and perhaps as strategic assets. But these assets are also critical resources for accomplishing broad strategic and learning goals. Making explicit how they contribute to a museum’s being a recognized convener for children’s well-being, supporting 21^st century learners or workforce development, or activating civic engagement around urgent social or environmental issues–is necessary to activating their value

The Benefits of Learning Assets

Beyond a general sense that identifying learning assets might be helpful, what does clarity about learning assets accomplish?

Earlier this year, I wrote about a hard-working question I often use as a tool: what can this attractive option accomplish, that needs to be accomplished, that another attractive option cannot? That’s a good test for evaluating learning assets as well. There are two perspectives on this question to consider. First what does this help the museum accomplish for itself organizationally and second, what does it accomplish for the audience?

Internally, defining a set of learning assets:

•          Creates a shared strategic and organizational context for all learning assets. Each asset has a distinct and significant contribution to make to the organization’s strategic and learning interests. Aligning assets on one strategic platform assures they all focus on advancing those interests.  

•           Establishes a framework for accountability. Responsibility for delivering learning value follows the asset by: calibrating organizational impacts to each asset and its capacities and opportunities; organizing staff for work; and allocating and managing resources.

•           Strengthens interdependence and interaction among learning assets. Clarifying that there are, in fact, learning assets and what they are makes each more valuable. Defining the contribution of each asset positions all of them for interaction by pointing out ways they can work with, support, and enhance one another.  

A museum’s valued and complementary learning resources help:

•           Serve the full audience and age range. Every learning asset does not have to serve the full audience or age range. Certain assets can serve targeted audience and age segments well. Together all of the assets can serve the audience fully.

•           Serve the audience with choice. Multiple assets increase the number and range of activities to assist learners in customizing their visit to interests, time, visitor group, and relationship with the museum.

•           Assist learners in extending and deepening their learning and play experiences. Managing learning assets to afford specific opportunities makes it easier for learners to build on an experience, follow-up on an activity in greater depth, explore interesting content through a new format, or experience a different perspective.

Working Assets

Defining a museum’s learning assets is a good start, but knowing their qualities and contributions is important to putting assets to work. I find four reference points helpful to consider: context, comparability, complementarity, and capacity.

1830's Log Home at Madison Children's Museum

•    Context explores an asset relative to the museum’s strategic and learning interests and to other assets. Consider the garden in relation to exhibits. What does it accomplish that an indoor exhibit does not? How important is the garden in providing first-hand experiences with naturally occurring seasonal cycles and changes important to the museum’s mission?

•     Comparability refers to the relative size of the assets. There can be wide variability among assets. In one museum a theater might be a program space for weekly performances; in another it might have a full schedule of live theater and demonstrations that explore current issues through science with students and the general public. Size is not only physical space or audience numbers. An asset may have staff assigned to it, a strong identity among constituents, or a recognized expertise for the museum.

•      Complementarity is the usefully different aspect of each necessary and valued resource through which a museum delivers learning value. Push on distinctions to probe what’s distinct and how it serves the other assets; a museum school is an on-going learning community with a continuity among students and teachers that programs do not have. Research on inquiry learning conducted in the museum's research center can be delivered through the teacher institute or programs; research on collections can inform exhibit planning.

Science Park at New York Hall of Science

•     Capacity is the asset’s ability to make an impact on or for learners or the community. Impacts­–or positive changes the museum believes are possible–are not easy to define. Still it’s helpful to keep a broad view of learning, be clear about the nature of the impact, and relate it to the organization’s long-term, intended impacts. How will the teacher institute reduce the achievement (or readiness) gap among students?

There are more ways to work a museum’s assets. Recently I heard from a science museum and library that I worked with 2 years to develop a learning framework that described 4 learning assets: exhibits; programs; collections and library; and the house and grounds. We also developed an exhibit master plan. The museum is now completing a program plan that describes the audience, vision, strategies, results areas, and impacts for programs. A collections plan is in the queue for 2012 and will be followed by an updated building and grounds plan. In addition to developing a plan for each asset that looks at and explores these assets at close range, the museum is using the 4 assets for budgeting and funding requests.

Your Museum’s Learning Assets

Size up your museum’s learning assets. Look around the museum to get a sense of where and how the learning assets are coming through on your website, in the organizational structure, the budget, and vocabulary. Think about the following.

•           How does your museum look at and talk about its exhibits, programs, outreach, film and   multi-media program, or resource center?

•            What are your museum’s leaning assets?

•           How do you define exhibits at your museum as a learning asset? or programs as a learning asset?

•           How does this help internally to strengthen these areas and encourage collaboration?

•           How has it benefitted your audience? your community?

Related Museum Notes Posts

• Learning Assets Part 1

A Museum’s Learning Assets - Part 1

Museums deliver learning value through exhibits and how many other ways? (California Academy of Sciences)

Exhibits and programs are the primary way museums and science centers deliver learning experiences and learning value.

And so are collections. So is film and multimedia programming. And also a school, a garden, or an historic building. What about a library? A teacher institute? Outreach or an aquarium?

Identifying the different ways museums are educationally valuable may start off easily enough. But the list quickly expands; more possibilities are found and more distinctions are made. If you’ve encountered this, you know it’s not easy. You should also know, it’s not uncommon.

Several years ago I developed a learning framework and exhibit plan with a museum that was experiencing multiple challenges. It re-opened at a new site just after 9-11 and it was undergoing multiple leadership transitions. Our planning process was a natural playing field for the tensions among exhibits, programs, and a museum/charter school the museum operated. In the course of planning, we needed to describe the primary ways in which the museum would implement its broad educational goals to serve its visitors, students, and community. It was tense with an impending sense of winners and losers; was the school really part of the museum? were programs being recognized for what they brought to the museum experience?

Four Small Words 

We avoided a seemingly inevitable hierarchy in what now seems like an obvious solution. We designated all three areas as valued and complementary assets through which the museum delivers learning value. In the end the plan’s legacy was not its pedagogical framework or exhibit concepts but the recognition that the museum needed all three of its learning assets. And they needed one another.

A family art program (Children's Discovery Museum of San Jose)

Identifying a museum’s learning assets also proved to be useful in developing a learning plan for 2-1/2 recently merged museums. Not only did each museum bring its valued assets to the new entity, but the new museum needed a single–and shared–set of learning assets to move forward. This approach has worked for a very established science center and museum with a planetarium and nature center as well as exhibits and programs; for a start-up children’s museum; and for an expanding science center.

Given their versatility and tested diplomatic capacity, learning assets have become a standard part of the strategic and learning planning I do with museums.

Learning Resources as Assets 
Learning assets serve as overarching objectives for accomplishing a museum’s long-term strategic or learning interests.  They may be referred to differently to fit a museum’s existing vocabulary. Some museums refer to them as learning platforms, others as learning resources.

The term assets, however, highlights useful dimensions for understanding and managing this accumulated value for a museum. Exhibits, programs, collections, gardens, historic structures, to name just some learning resources, are financial assets. Some, such as exhibits, are significant sources of revenue. Museums also invest in them on an on-going basis: hiring and supporting staff, developing expertise, and making new acquisitions.

The learning value of exhibits, gardens, a library, or a school should be considered as well. They attract audiences and engage specific visitor segments in targeted ways. They carry content, cultural heritage, and community stories. They embed the museum’s experiential brand and distinguish it from other venues that serve the same audiences.

Learning assets clearly also reflect museums as multidimensional learning and community resources. Although virtually all museums have exhibits, they are not just a bunch of exhibits. A planetarium is not just a “component;” programs are not simply a service. Thinking of them narrowly limits leveraging their format, media, content, or type of interaction to engage learners in pursuing interests, practicing skills, appreciating their heritage, or building a sense of community. In fact, when a museum views its exhibits and programs as a set of learning assets, or perhaps a portfolio of assets, it has a greater opportunity to impact learners and the community.

Defining Assets

Mill City Museum (Minneapolis): Building as artifacts as learning asset

After quickly deciding that exhibits and programs are learning assets, determining what else should be included can take some time. Lively discussions are ahead for the museum that explores whether its building is a learning asset as it is for The Bakken Museum or might be for the California Academy of Sciences. Could Lake Champlain be a learning asset for ECHO Lake Aquarium and Science Center? Should a museum’s website or social media be a learning asset? 

To identify assets and understand why they are valuable, ask: how does this resource advance our museum’s learning and strategic interests?  What are the particular learning interests it advances and how? Be specific, be concrete; give examples; back up with as much evidence as you have.

While the intent of this exercise is not to eliminate assets, it's possible there will be some reshuffling. Is the garden an exhibit or part of the house and grounds? Is science park an exhibit?  It's also possible the more limited value of some resources may become apparent to everyone; what is the recycling center accomplishing for us? This process may also help a museum recognize it has more learning assets than it thought it had.

Garden or exhibit? Greensboro Children's Museum's Edible Garden   

So far I have mentioned 15 various learning assets (in blue) that I know of in particular museums, although no one museum has all of them. Quantity aside, what counts for a learning asset in one museum may not register as one at another.

There’s no recipe for describing learning assets. It's important, however, to define them in parallel ways, emphasize what distinguishes each, and insist on the valued and complementary nature of these assets. An example for exhibits and for programs illustrates brevity, parallel construction, and distinctions.

Exhibits engage children and adults in self-directed, shared learning experiences through hands-on, bodies-on, and minds-on interaction in rich, experiential settings. 

Programs provide children and adults with special access to media, tools, objects, and processes through focused and facilitated experiences by prepared staff, volunteers, artists, scientists, and specialists.

One of several libraries at Seattle Art Museum

Both these learning assets mention the same audience. This isn’t always the case; a research library might serve local, national, and international scholars. Each asset does distinguish itself in the nature of the engagement it offers. Exhibits offer self-directed experiences while programs offer facilitated, or face-to-face experiences. Facilitated experiences allow special access to fragile objects (or hot glue guns) whereas the rich experiential settings of exhibits encourage physical interaction. These descriptions reflect the museum’s mission in general, although, they could do so more explicitly.

Definitions vary in length from one museum to another as does the territory they cover.  What's important is being clear about the capacity of different assets to deliver value and contribute to a cohesive set that works together. 

So, once you get this far, then what? Join me next week. I'll look at benefits of using learning assets.

Related Museum Notes Posts

• Learning Assets, Part 2

Wonder Years: The Story of Early Childhood Development

Wonder Years: The Story of Early Childhood Development, an exhibition from the Science Museum of Minnesota (SMM) should be in a mall. I mean that in a good way and as a sincere compliment.

The exhibition presents the science of young children’s development and insights into their everyday learning from the world around them and places both in a larger social and civic context. A clear long-term community interest is expressed in an ambitious goal: “Ensure that all children benefit from the growing body of knowledge about the science of early childhood development and get the best start in life.”

While the exhibition works in the Museum’s 4^th floor Human Body gallery, a mall setting would bring this message to a broader audience and further build on the Museum’s mission: Turn on the science: realizing the potential of policymakers, educators, and individuals to achieve full civic and economic participation in the world.

 My interest in the strong children-strong communities connection I’ve written about previously drew me to explore Wonder Years three times from March to late June. The first time I participated in an SMM tour for Minnesota Association of Museums and heard from Project Leader, Laurie Fink, about the exhibition. I spent an afternoon on my own exploring Wonder Years and a third time, I ducked in to it with a friend prior to visiting Tutankhamun.

SMM and its partners have managed the challenge of creating an engaging visitor experience with a serious public message. Alignment of a solid conceptual approach with its intended audience, a varied experiential mix, and a robust community platform is deliberate and strong.

The exhibition and related programs and events is a partnership among SMM, the Center for Early Education and Development (CEED) at the University of Minnesota, and Public Agenda, a non-partisan engagement organization. Funding is from the National Science Foundation.

Reaching a Broad Audience

A Wonder Years flyer identifies the intended audience as teens and adults, parents and non-parents, anyone with a child,  was a child, or lives in a community with young children. In short, the audience is intended to be everyone. On each of my visits I saw a variety of visitors and visiting groups. There were 3-generation families, parent-child pairs, families with young children and with children of very different ages, groups of teens, groups of young adults, and young couples.

Some of these and future visitors may also be policy makers and community leaders. Very likely, many adults in the exhibition have a major role in the lives of young children–their own children, grandchildren, nieces or nephews; young neighbors, students, or youth volunteers. These relationships provide strong personal connections to exhibition content. As a neighbor, volunteer, mentor, caregiver, voter, and taxpayer, virtually every adult has a citizen’s interest in children’s current well-being and potential. Children’s experiences today shape their capacity as adults to work and take part in the community; this is the source of a community’s future vitality and resilience.

The flyer also makes clear that Wonder Years intends to reach and engage members of the community more broadly through public programs as noted below. These will undoubtedly draw in members from other parts of the community. Yet, broad and varied as visitors to SMM are and as inclusive as related programs are, a mall would undoubtedly provide a broader audience.

A Solid Conceptual Approach

The exhibition brings together recent research on early brain development, insights into how young children learn from the world about them, and a perspective on early childhood development as a common interest. 

A positive view of young children as strong, competent, and born to learn comes through clearly. Focusing on children's remarkable accomplishments before they even enter school and highlighting the great promise of the first years of life, the exhibition does establish early childhood as the Wonder Years. Overall it avoids clichés about young children and presenting them as cute. Large images of children on photo banners show them playing, engaging in everyday exploration, talking, touching, and laughing, hugging and being held. These images express and reinforce messages about the importance of positive relationships, active engagement with the world, and early language experiences. Occasionally the gleeful laughter of a laughing baby video can be heard and it is delightful.

Four key messages organize exhibition content and guide lay-out. The interests and perspectives of researchers, experts, parents, young children, and citizens intersect with areas and activities and reinforce the messages.

•             How scientists learn about children’s development looks at pioneers of child development research, how scientists study young children, and some questions that interest researchers.

•             What is happening in early development explores the role of early experience in shaping the brain and brain architecture; early language experience and the brain; and early social, emotional, physical, and cognitive development milestones.

•             How young children learn from the world around them highlights some of the amazing feats of early learning, the role of vision in language development, and the importance of play.

•             The impact each of us has on the healthy development of our youngest citizens brings together the impact of strong and positive relationships, the importance of everyday moments, and the contribution of many hands to the experiences children need for the best start in life. 

An important principle in early development comes across consistently throughout the exhibition but it is not made explicit. Interaction is critical and constant: interaction among developmental domains (physical, social, emotional, etc.), between children and the environment; and  interactions with parents, other caregivers, and others involved in children's lives.

Experience As Message

Wonder Years takes on and meets challenges many exhibitions face in translating information rich topics into active experiences without compromising content. It covers recent research, technical information on brain development, and hard to visualize processes of language acquisition and development. Resisting the temptations of extensive text, Wonder Years forges a fresh mix of familiar formats and creative strategies to present content experientially.

Videos, biological specimens, microscopes, quizzes, puzzles, and games familiar in many exhibitions are used well here. There are opportunities to hear what experts say, set-ups for conversation, and an invitation to record opinions on issues. Content is accessible and presented in both English and Spanish in text and videos. The writing style is generally clear and engaging and sometimes crisp, like “Experience is the brain’s electrician.”

How pruning makes the brain more efficient

At components like Connections Shape the Brain, text, images, and a relevant instrument or model work smoothly together. Illustrating how pruning makes the brain more efficient, flashing lights show the path a message might take as it travels through the brain of a toddler, a child, and a teen. A mother comparing the three ages told her 5-ish year old son, “That’s’ why we do things over and over again. So we can do them faster.”

What the exhibition does exceedingly well is to power exhibition messages with visitor activity, sometimes getting inside the experience of a researcher, a parent, even a baby. This parallels the critical importance of experience for early brain development; “Experience shapes the brain,” one text panel notes. Similarly, engaging in conversations, tasks, role-play, and perspective taking affords a first-hand experience.

"We are going to do an experiment."

 Children’s activity sometimes becomes the exhibit. I saw parents test their child’s “theory of mind” or being able to understand that others have their own thoughts and feelings. Saying, “We’re going to do an experiment with you, Kayla.” a mother settled in at the table and guided her 2-year old daughter through the test. 

The child's activity becomes the exhibit.

In the exhibition’s pretend kitchen, children climbed on the stool at the sink, turned on faucets, and washed dishes. Well-positioned text panels cued parents on opportunities for language, interaction, and how these support a child’s development for precisely those child behaviors they might see. Nearby, children as well as adults looked through a set of three goggles that allowed them to see like a baby sees at birth, at 1 month, and at 3 months.

Seeing as newborns and infants see.

The exhibition design is not as strong as the content. Occasionally it detracts from the experience and messages. Multiple exits and entries interrupt the flow serving as paths between the building's elevators and stairs. White text on pale blue panels renders text almost unreadable. One video is mysteriously placed at about 8 feet above the floor. Finally, design styles, particularly color palettes, differ between areas and feel as if they came from separate designers or even projects.

 

Community Platform

Wonder Years is framed as a project with an exhibition. Its strong and complementary partners are key to this configuration. They combine expert content, experienced public engagement, and a recognized public venue.

A slate of community programs has significant, although less obvious, presence than the exhibition–certainly for museum visitors. Nevertheless, they are planned to actively engage members of the community in considering what the science of early development means to us as individuals, to our communities, and to our youngest citizens.

• Citizen Conferences were held in the Twin Cities and in Greater Minnesota in May 2011. For parents, early childhood advocates, state and local policymakers, and interested community members and citizens, the focus on these half-day working sessions was to explore society’s responsibility to children birth to 5 years.
• Issue Conventions are intended to encourage smaller groups of organizations and community groups to explore how the exhibition and programs can further their efforts at educating others and expanding networks.
• Public Forums in Fall 2011 will bring experts and community members together for public presentations and conversations on the scientific findings and their implications for families and communities.

The partnership mix also suggests how seriously SMM is taking its goal for all children to benefit from the growing body of knowledge about the science of early childhood development and to get the best start in life.

Wonder Years is an engaging and responsible look at a critical social issue: the early development of children, their future capacity, and the ultimate strength of our communities. I think this makes Wonder Years a great exhibit to see at the mall. I hope you get to see it there. 

Resources

•            Gopnik, Alison; A. Meltzoff and P. A. Kuhl. (1999). The Scientist in the Crib: What Early Learning Tells Us About the Mind. William Morrow & Co: New York.

•            Hart, B. and T. R. Risely (1995). Meaningful Differences in the Everyday Experience of Young American Children. Paul H. Brookes Publishing Co: Baltimore, MD.

•             Schonkoff, Jack P. and D. A. Phillips (Ed.) (2000). From Neurons to Neighborhoods: The Science of Early Childhood Development. National Academy Press: Washington, DC.