Social Network Analysis and Evaluation: Learnings From the Evaluator and the Client
This article is rated as:
This past year, our team at Three Hive Consulting worked with a community development initiative to evaluate their activities. A key function of the initiative was to serve as a connector and integrator, bringing together different individuals, groups, and organizations who were all interested in strengthening the lives of the citizens in a specific community.
One of the goals of the initiative was to create community linkages and foster collaboration. We chose social network analysis (SNA) as the methodology to understand how the initiative had affected the relationships and collaborations in the community.
Before we jump into what we learned, and what the client thought, let’s back up and review what an SNA is and how our team used the methodology for this client.
What Is A Social Network Analysis?
An SNA is a quantitative approach to measure the strength and types of relationships within a network. It examines how different players relate to each other and assesses the characteristics of those relationships.
An SNA focuses on network structure, rather than the individual characteristics of the partners.
Data about who is included in the network, how they are related, and how they are working together is collected to create a picture of the relationships in that specific network.
What Is Social Network Analysis Used For?
An SNA can be used in an evaluation to measure how groups of people are working together to achieve a common goal. It’s a particularly useful tool when addressing complex issues where multiple players are working towards a common outcome.
When the functioning of the network can impact whether the group achieves its goal, an SNA analysis can uncover insights about how the players are working together and pinpoint areas for improvement- where relationships can be strengthened or built.
A Social Network Analysis is also useful for community development or population health initiatives where the actions of a group of players contribute to change.
Social network analyses can help to answer questions such as:
Who is connected to whom?
How well are the existing relationships working?
How connected are different sectors within the network?
How strong are these connections?
Do these connections have directionality?
Who are the central players and outliers?
Where do relationships need to be built or strengthened?
How is power shared within a network?
Who shares resources with whom? What kind of resources?
How Did We Use Social Network Analysis?
One of the first steps in a social network analysis is bounding the network; which means deciding who is or isn't considered part of the network.
In our case, the evaluation sub-committee, which was made up of community partners and staff from the initiative, listed who the initiative was directly working with and categorized each partner by sector (e.g. faith, education, recreation) and by the type of community capital they provided (e.g. natural, built, social, financial).
We used the PARTNER Platform by Visible Network Labs to send the survey and analyze the data.
Although the use of the PARTNER platform simplified the process, gathering the data was still time-intensive for all involved to achieve the high response rate (>75%) required for data accuracy.
While using the PARTNER platform also simplified data analysis, sense-making was, again, time-consuming.
The data from the survey was complex. We as evaluators looked to find the overarching trends in the data, then worked with the evaluation sub-committee to delve into the nuances and understand what the data meant to them.
For example, we generated a map of which partners were sharing resources with each other (see graph below). The colour coding represented the partner’s sector.
The SNA generated many maps like this, which helped the client visually see which organizations were connecting with each other and who was being left out. However, when it came to understanding what the map meant for contributing to the initiative’s success, we needed more context.
We presented these maps to the evaluation sub-committee and together we worked to make sense of the data. Some maps and insights were more relevant and inspired ideas for action based on who could be collaborating more or which two unconnected partners might work well together.
We also generated value, trust, and connectedness scores for the network as a whole and for each individual partner.
When we compared each partner’s degree of trustworthiness with how well connected they were, an interesting pattern emerged. Visualizing these results allowed us to identify partners who were highly trusted but not well connected (black dots in the graph below).
Again, it was up to the evaluation sub-committee to put these results into context, using their knowledge of the partners to decide this was an expected result or whether connecting these highly trusted but not well-connected partners was feasible and relevant to achieving the network’s goals.
In some cases, a highly trusted but not well-connected partner may only be involved in the network’s work tangentially and it wouldn’t have made sense to connect them with more organizations.
To learn more about using the results, we asked our client how the social network analysis was useful to them.
Client Perspective
About the overall experience:
"Being invited to design and create a Social Network Analysis was a meaningful experience. It was helpful to consider different variables of our network, that perhaps had only been tacitly considered – like value and trust. Or, we had considered these things, as they come up regularly in partnerships, but we hadn’t formalized the concepts or explored how they could be measured. Learning more about this was insightful and empowering."
About visualizing the connection in the network as a map:
"Practically, I had a visual understanding of our network and saw right away that meaningful connections needed to be made between some of our partners. Having partners collaborating with us does not mean that they are collaborating with each other, or even aware of each other. We have used this insight to connect more people to [one organization], as well as linking some of our partners from the business sector into what have traditionally been agency-dominated conversations exploring food security. "
About using the trust and value scores:
"In some ways, the obvious is profound and humbling. We were well connected, in our network, but we were also shown to be trusted and valued. This was immediately encouraging and affirming, especially when considering that we haven’t always delivered, or moved forward with clarity, or met the expectations of all stakeholders."
About using the recommendations:
"The recommendations in the final report have been very valuable and will inform strategy, especially over the next 11 months...[for example] we are working to improve our communication with partners and employing a number of different methods. We have intentionally started... to share information and to host projects and groups. It is working."
What we Learned From Conducting A Social Network Analysis
A social network analysis must be a collaborative effort
Everyone needs to be on board with taking a collaborative approach to the project; from deciding who is included in the network and how to categorize them, to encouraging partners to participate in the survey, and understanding the data in context.
Interpret the data with the client to make it more meaningful
Understanding how groups work together requires an understanding of the context and a level of detail that an external evaluator doesn’t have. The data only became useful once we started asking the client what it meant to them. If a client isn’t willing to work through the data and generate insights and recommendations together, the SNA loses much of its value.
Your social network analysis will take time
The biggest lesson here was how much time and effort it takes to conduct a meaningful social network analysis. It takes communication and follow-up with network members to get buy-in and a high response rate on the network survey. Analyzing the results and making sense of the data must be done in conjunction with those involved in the network.
Clearly defining your network is extremely important
Without a clearly defined network, an SNA can quickly become bloated and confusing. A network should have clear inclusion or exclusion criteria and include a clear definition of the goal the network is working towards. An SNA may not be very useful when a group of organizations or people are not working towards a specific goal.
To learn more about how the evaluation approach you select affects your evaluation questions, check this out.
To learn more about applying evaluation in practice, check out more of our articles, or connect with us over on Twitter (@EvalAcadmey) or LinkedIn.