Organizational Capacity Building

Candace LaRue and Associates

Promise neighborhoods panel live blog post #CICSUMMIT

Michael McAfee

Measuring Performance for Promise Neighborhoods: Collecting Data and Reporting Results

Promise Neighborhoods is a current opportunity to work on a core set of results over a period of time. It is grounded in 10 results and 15 indicators. The implementation grant competition is built from a results framework. Indicators are focused on education, communities, and supports. The intention is to use the neighborhood as a unit to advance guiding principles: working at scale (serving all of the children in the neighborhood) and creating a culture of accountability. Many of these principles are similar to the Harlem Children’s Zone. There are right now 5 implementation Grantees working over 5 years. All PN Grantees are following a cradle-college-career strategy. He do we weave systems of support to address the complete continuum. We understand what data will tell us if we don’t learn from past experience. So what? PN is not a new idea, nor is it a silv bullet. It should not be divisive in your community. promise Neighborhoods can hold all of that. If we are doing good cradle to career work, we will be building organizations that can deliver. It is easy to ask for results and to move indicators, but it is hard to ask if we are insetting in organizations to be able to deliver.

The framework: define the neighborhood, identify partners, develop contracts for accountability, capitalize on partnerships to really deliver over the long term. It is difficult for organizations to fund administrative capacity such as data system and days system management. It is important to think about what it really takes to do this work. W have to include all the stake holders, including children, parents, teachers, service providers, and policy makers. It is important to think early on about how each partner will contribute. How do non-profits do the work? How should funders be funding it? And how can policy makers remove variables? There is nothing special about PN…it only becomes special when we do something about it. What makes it special is what we do with it in communities.

The PN institute at PolicyLink was established to provide technical support to PN Grantees. This includes getting the federal appropriation every year, and possibly to make it permanent. They are working to build communities of practice by bringing Grantees together at least 3 times per year. We provide technical assistance – one on one coaching, reports, gathering resources together. How do we build national infrastructure so we can mature as a field? We should not be building new databases and governance structures every time there is a new opportunity. We should be building capacity so these things don’t have to be recreated. Some communities were struggling with data systems, especially those who did not have high capacity regarding data. PNI purchased licenses for Grantees from Social Solutions, so they could hold some of that capacity in their organization. The data system has allowed there to be a common conversation. The data needs to be aggregated to track data at the national level. The PN communities are using a common data platform, working on the same results and indicators, and using the same framework for action. How do communities develop the capacity to do this work at this scale? Some of the capacity needs to be held as national infrastructure. A cradle to career strategy suggests a 20 year commitment – but where is the 20 year investment? If we can invest in PN strategies over the long term then we can achieve results and scale up the work that needs to be done to move an indicator, and build the capacity of organizations so they can move beyond failed models. In congress, people have read all about what non-profits want to do, but where are the results? This is an opportunity to build capacity, to learn to have different conversations, and move beyond this. Promise Neighborhoods is a funding stream and strategy, and it should be used with all the other good work being done in a community.

Jennifer Comey: Data Technical Assistance for Promise Neighborhoods

She is part of the National Indicators Project in DC, and has worked with the DC Promise Neighborhood initiative for 2 years (now moving forward to implementation). The Urban Institute was awarded a contract to provide national technical assistance.

Goals of presentation:
– review efforts to assist PN implementation sites with data definition and data collection
– describe required performance measurement for PN
– Lost potential sources of data
– challenges for data collection and evaluation (place-based initiative)

Task 1: data definitions
– common data definition and methods of collection
— project and program indicators
— implementation indicators
— neighborhood indicators
— various levels of observation: individual, school, and neighborhood
– technical working group

The department of education did not provide a data system or a specific set of requirements around data indicators. The Grantees were able to identify what they thought about data collection, and the UI worked to create commonality among Grantees. Grantees were required to set up a longitudinal data system, which became a case management system. Grantees also need to understand how the neighborhood is changing over time.

20121115-111307.jpg

Goals
1. Enter kindergarten ready
2. proficient in core subjects
3. successful transition from Ms to HS
4. graduate from HS
5. high school graduates obtain post secondary degree, citric action, or credential (without the need for remediation)

20121115-111533.jpg

Goals
6. Students are healthy
7. Students feel safe at school and in their community
8. Students live in stable communities
9. Families and community members support learning in PN schools
10. Students have access to 21st century learning tools

Potential Sources for GPRA Indicators
1. Individual administrative data
2. Aggregated administrative data
3. Neighborhood survey
4. School climate survey

Urban Institute will be publishing a guidance document. At the beginning for UI, the goal was individual level data, but that wouldn’t necessarily allow Grantees to measure the broader neighborhood rather than just people participating in programs.

Case Management Data System
– track participants, services received, and programs on goals
– coordinate and communicate between programs and solutions
– determine effective solutions and make changes

All PN sites are a partnership of multiple organizations, state, local, and non-profit. It is very difficult to create a unified case management data system due to confidentiality, sensitivity, etc.

Task 2: National Promise Neighborhood TA
– TA for sites to collect GPRA indicators and other programmatic performance data
– TA for local data systems

This includes TA on issues of surveys, IRB issues, etc.

Task 3: National PN efforts
– collect grantee data and prepare are restricted – use data files for researchers
– developing code book and documentation of files
– risk disclosure analysis

Task 4: National PN Efforts
– annual performance reports
— develop databae to collect GPRA indicator data
— TA for forms
__ analysis of baseline data and eventual performance in GPRA indicators in aggregate

Challenges
1. Tension between place-based vs project-based initiative
2. era of school choice
3. people move – do the successful families succeed and move out?
4. penetration rates are expected to increase
5. formative evaluation as opposed to a random experimental design

Alison Churilla and Paul Mattessich: Promise Neighborhood, Strive, and MN compas: Lessons from the Nascent Implementation of Data-Driven Cradle to Career Initiatives

Michael started with a conceptual framework, and Jennifer presented a methodological framework. How do you get this to work in a community? How do you work with multiple languages, or competing political interests, or when people believe the computers can do everything for them? wilder has worked with 2 Promise Neighborhoods projects and a STRIVE project, navigating these challenges.

20121115-113040.jpg

Data is meant to tell a story, and MN compass is more than just an indicator project – these are people, individuals, and communities. MN compass provides unbiased, credible information curated by the topic area advisement groups. For example, there is an advisement group for Education with a variety of stakeholders to tell them what information they need to do their job and improve life. MN Compass provides a set of key measures that tell an initial story and the “tip of the iceberg” on these measures. The data is “cut” by different demographics, I.e. race and ethnicity, age, gender, and income. MN Compass also provides additional resources/information – I.e.evidence-based programs.

Information is kept as up to date as possible, as a we based tool – MN Compass

There are three types of PN indicators – Community Outcome Indicators, Promise Neighborhood Outcome Indicators, and Process indicators

COI – education level of workforce, crime rates, property rate, etc
PNI – achievement scores, graduation rates, etc
PI – use of navigators, participation in out-of-school time activities, etc (longitudinal database)

Putting together a team that can gather all these types of data require different skill sets.

Creating indicators is science, politics, and art.

Note: these are my notes from the presentation, not my opinion. In the case of Jennifer Comey, the notes follow her power point very closely (as did her presentation).

Advertisements
Leave a comment »

Lyle Wray measurement framework and the evolution of our field #CICSUMMIT

A conceptual framework is the basis for many theories. We have implicit ways of organizing data. It is a group of concepts that provides a focus and a rationale – what is important and why is it important. We are drowning in data. On the train ride up here, everyone was looking at a screen. How to you integrate these massive amounts of data? How do you know this is important? We develop our frameworks through word models.

The Thomas Acquinas used a framework to make room for science 800 years ago. They have a variety of domains (economic, environment, etc). An example is sustainability – that has a very broad framework. Frameworks are nested from local to broad. We need to figure out indicators that next from a neighborhood to the world. Instead we have a Tower of Babel where the data doesn’t work well together. It is important to be able to both nest and disaggregate data. We have incompatible layers. Frameworks have different time frames – some things change quickly, some change slowly. For example, dashboards are short term but reports are long term. Frameworks may be based on individual measures, portfolio measures, or composites. Portfolio measures may be collapsed into an index.

Domains – from simple to direct. Some measures are direct measures (like birthweight) and others are complex and based on theoretical judgment. For example, if someone gets shot, it increases the GDP – what is wrong with this picture? We need a broader measure of social well-being, like the Fordham measure. An example of nesting is being able to understand local sustainability in a global context.

Composites or singular indicators? It depends on what we want to measure. Higher level indicators are helpful to communicate. If you don’t have them, you are “down in the weeds” and can’t see the overall effect. But, then the issue is the selection and weight of elements in the index. What is the decision process for developing the measures? An example is the Legatim Prosperity Index created for France by Stiglitz and Sen. The US 12th in this index. You can “drill down” to individual elements and countries and then aggregate up to the world and the composite index.

The triple bottom line is another framework – economic performance, social performance, environment performance.

Another example is the Paul Epstein cause and effect chart on childhood obesity. They identified six major causes of obesity – prenatal practices, genetics, early feeding, environment, lifestyle, and policy. The cause and effect chart provides a way to identify an intervention approach on what might be more “moveable.” This is in contrast to the “intervention du jour.” Unless you have an understanding causes and effects, it is easier to “peddle” interventions that do not have evidence to support them.

Another example – the Balanced scorecard framework (see Kaplan and Norton 1996). This gives multiple perspectives that need to be looked at at the same time – financial, customer, internal business processes, learning and growth. Vision and strategy is at the center. One perspective is not enough. Learning and growth implies that organizations need to get better all the time, not have the same problems all the time. Internal business processes ask how much it costs the business to “get it out the door.” The customer perspective highlights customer relationships. This framework goes beyond the financial perspective, but organizations that have adopted it do better than those that focus on financials only.

What are the benefits of frameworks? Analysis – comparison across communities, analysis and synthesis cycle, scope and scale, point to prospective indicators. Where are we, what are we doing, and what do we know? Communication – common language, points attention to important elements (example of checklists for important repetitive tasks like preparing for surgery), improves communication, organize to understand 5+- 2 (this is how much we can understand at one time). This allows us to communicate with a common language while avoiding starting from scratch. Action – point to intervention targets, avoids intervention du jour, select promising practices based on comparison data (omega squared – powerful variables, not the ones that you have to tease and twist to identify impacts). How do we actually get things to change? We need to use our frameworks to understand what we can do to create change. We need Big Bang variables in order to move achievements of struggling students faster. Frameworks can help point us to the game changing variables.

Frameworks take a while to develop, and there are many evolving frameworks. We need to integrate nested frameworks into the massive among data we have coming down the tubes. Who convenes and how to we make progress on frameworks over time?

Note: these are notes from his talk, not my own opinion.

Leave a comment »

Charlotte Khan – data in the context of national and global trends live blog post #CICSUMMIT

Local communities are operating in a global context, and it is important to include that context in our understanding of what is occurring at the local level.

There is an unprecedented rate of change right now, with, for example, population growing more rapidly than ever before. The majority of population growth is in Asia and Africa.

Another important trend is automation in technological change. Today 3 billion people are chasing 1.2 billion jobs, making educational attainment even more important to “outrun robots and computers.”

In 2010, a report came out on global reading scale scores. The USA was 17th, but unpacking by race and ethnicity, White and Asian students were towards the top of the list, but Black and Latino students ranked in the 40s. This achievement gap is not acceptable.

How did we get here? Betwee 1947 and 1979, there was broad based prosperity. People at the bottom of the distribution benefited most. Since then, productivity has increased but wages have been essentially flat. Inequality has widened, and people at the top have benefited from growth but people at the bottom have suffered (note – this is partially because gains in productivity from technology are attributed to the capital, owned by the wealthy)

In the last decade, corporate taxes have gone down, profits have gone up, and job growth/creation has been going down. Moreover, the economy has been incredibly volatile with a rapid boom and bust cycle. The US economy is 70% dependent on consumer spending. In 2007, there was record- low savings and record high spending. By going to school, getting an education, and buying a home, people ended up in debt. When the housing market softened, we fell off an “economic cliff” into the Great Recession. What we need to understand in terms of our communities is that this recession has created a widening of inequality in the nation and in communities. Latinos, Asians, and African Americans have all lost more assets due to the housing bust than Whites.

This last recession has popped a bubble we were all living in, and “nothing will ever be the same again.” We need to be creating new jobs that will create sustainable wealth for families.

An example of this “longest bubble” is Health Care, with health care costs going up much faster than wages. And yet, even though we are spending so much on health care, obesity has doubled between 1995 and 2007. Our health outcomes, also, are lagging behind other industrialized countries, including life expectancy. Health care spending is also crowding out other investment – health care costs rose 75% in the MA budget from 2010-2011. We are waistline about 1/3 of health care expenditures on unnecessary or harmful health care expenditure – in MA, eliminating that expense would more than make up for the Budget Gap.

The US has the highest level of income inequality and lowest mobility rate among our peers.

In comparison with our wealthy developed peer nations, Americans are: the most personally indebted, incarcerated, unequal, obese, pay the most for health care, most energy, least inter generational mobility.

The young population is increasingly made up of people of color. If we don’t fix achievement gaps, we are in trouble. Cities in the Us are increasingly vulnerable to climate change, such as flooding an drought.

We need new ways of working. We need shared indicators, regional alignment on charged goals and a civic agenda. We need to use open source data analysis and visualization software, such as Weave, so we can communicate data to our communities and take advantage of the open data movement. We need to help our communities to put data together in such a way that the data can give them a handle on what is happening and traction on what the solutions can be.

Please note: these are my notes from her presentation, not my own opinion (except for where I started the statement with “note:”)

Leave a comment »