Earlier this month I helped facilitate a workshop at ETH CC covering the Impact Reporting section of the Grants Program Canvas. Check @Viriya’s post here for more info about this excellent new resource for the community!
We had a very lively discussion. Here are some notes I took:
Prompt
If we want to 10-100x the grant funding happening on crypto rails, then we need to prove that these mechanisms are more impactful. Not just 10% better, but 10-100x better. Impact measurement can’t be an afterthought that the intern does while everyone else is on vacation in August. It needs to be the selling point.
Let’s discuss:
- Why should we care about this in the first place?
- How to do retros and get 360 feedback on rounds?
- How to capture both quantitative and qualitative data about impact?
- How to get interim reporting from grantees and verify accuracy?
- How to get case studies and testimonials about long-term impact?
Why should we care about this in the first place?
- Altruistic: Build in public … fail in public too … let others learn from your experiences
- Selfish: Your reports become your case studies that power your marketing
- Legitimacy: Proof that each funding round tests a hypothesis, learns from it, and strives to improve the next round
- Competitive advantage: Grants = growth
How to do retros and get 360 feedback on rounds?
-
Table Stakes
- Announce the Results: Summarize how much was distributed and to how many projects.
- Publish a Post-Round Summary: Release a forum or blog post within a month after the round ends.
-
Gold Standard
- Comprehensive Surveys: Conduct 360 surveys with grantees and voters, focusing on both operations and impact.
- Hypothesis Reassessment: Revisit and evaluate the hypotheses stated at the beginning of the round.
- Grantee Pulse Checks: Understand the performance and health of grantees at regular intervals.
How to capture both quantitative and qualitative data about impact?
-
Table Stakes
- Self-Reporting: Track activities and progress against milestones.
- High-Level Metrics: Consolidate metrics on activities and outputs of projects.
-
Gold Standard
- Decentralized Reviews: Incorporate decentralized reviews and attestations to enhance credibility.
- Outcome Analysis: Consolidate comparable metrics on outcomes and deeper impacts of the funding.
- Funding Effectiveness Analysis: Conduct a deep analysis of where the funding went and its effectiveness.
- Social Proof: Analyze who backed projects early (wisdom of the crowds/social graph).
How to get interim reporting from grantees and verify accuracy?
-
Table Stakes
- Regular Updates: Require projects to post regular updates on a public forum.
- Milestone Verification: Check if projects have fulfilled their stated objectives.
-
Gold Standard
- Public Accountability: Utilize tools like Karma GAP for milestone verification, third-party reviews, and consolidating project reputation.
- Passive Monitoring: Implement passive monitoring via work artifacts (eg, GitHubs) to continuously track progress and outcomes.
- Third-Party Reviews: Do audit bounties, spot checks, or other independent assessments of activities and impact.
How to get case studies and testimonials about long-term impact?
-
Table Stakes
- Social Media Compilation: Collect and compile relevant tweets and other social media mentions into a blog post to showcase impact.
- Bloc Posts: Do interviews and turn them into blog posts or short form features.
-
Gold Standard
- Longitudinal Studies: Conduct longitudinal studies to track the long-term impact of funded projects.
- Counterfactual Analysis: Perform counterfactual analysis to understand what might have happened in the absence of the grant funding.
- Comprehensive Testimonials: Gather detailed case studies and testimonials from stakeholders other than grantees (eg, users, beneficiaries, etc) to illustrate long-term impact.
Related threads
A few other things since ETH CC that could be of interest to this group:
- @eleventh19 Tweet thread on how grant programs need to get the basics right: x.com
- UBI study: Findings | OpenResearch
- OSO’s assessment of Retro Funding impact