Newest proposal: https://gov.gitcoin.co/t/proposal-fdd-season-14-budget-request/10658
Workstream Mandate: Defend Gitcoin (Grants, DAO, & future products) from threats to its legitimacy, credible neutrality, and sustainability.
Season 14 Budget Proposal
This is a renewal proposal for the Fraud Detection & Defense workstream requesting budgetary funds for Season 14 (1 May 2022 through 31 July 2022).
FDD is requesting $486,130 to fund Season 14 (S14) and replenish its reserves for S15.
High Level Budget Breakdown
Season 14 Budget Total = $649,455
PLUS Season 15 Reserves = $432,970
Total Needed = $1,082,425
MINUS Current Reserves = $596,295
($690k total holdings | $596k Season 13 reserves & $94k treasury)
Total Amount requested at this time = $486,130*
*The GTC total will be adjusted based on the current market value at the time this proposal is moved to Snapshot and to Tally. The vote will be to send $ in GTC at the current price when moved to Tally or the 20 day moving average (whichever is lower).
TLDR
FDD is responsible for the following items which are mapped to the outcome sets below
- Defending Gitcoin Grants from exploits. (Sybil Accounts, Collusion, etc)
- Coordinating the grant eligibility policy & reviews
- Sustainably operating the workstream. Goal setting, delivery, accountability, & comms.
Sybil Defenders (Legitimacy)
- Maintains DevOps for Sybil Detection machine learning pipeline
- Designing models for decentralizing the inputs to the machine learning pipeline
- Humans evaluating accounts to determine if they are sybil and train machine learning
- Solving for sustainable mechanism design minimizing the exploit surface area/profitability
- Inform decisions and improve our resilience to fraud using research, simulations, NLP/ML, etc
Grants Intelligence Agency (Credible Neutrality)
- Set up tooling to lower our cost, improve the quality of reviews, and give transparency/consistency to the decision processes
- Execute all grant reviews, disputes, and appeals to ensure grants rounds run fairly
- Design, test, and implement a model for decentralizing grant reviews using incentives
- Deep dive reviews of grants to better understand the methods attackers use
Evolution (Sustainability)
- Ensuring governance decisions and processes are properly designed and implemented
- Supporting payments and advancement of smart contract usage in comp models
- Articulating the work being done in FDD for internal and external stakeholders
- Setting strategy goals and ensuring that the workstream is accountable for delivering
- Providing data analysis services for FDD (this season we are opening to all GitcoinDAO)
Season 13 Recap
Key highlights & learnings:
Sybil Defenders
- GitcoinDAO now runs the full anti-sybil operational process (previously done by Blockscience)
- Doubled the number of human evaluations done at â…“ of the cost
- Classified sybil behaviors & ran impact analysis
- Performed trust bonus analysis and provided recommendations
- Won Metapod pitch competition and validated users for Sybil Detection DAO
- Reviewed sybil assumptions and began work on mechanism design & simulations
Grants Intelligence Agency
- Hit all primary round OKRs without getting data integration on time from Holdings
- Assisted PGF ops in tag requests for eco & cause rounds
- Began creation of “trusted seed” of reviewers
- Designed curation incentivization structure for any community to have community curation
- Grants Curation NLP model that can replicate the GIA team’s decisions with 69% accuracy
- Reviewed and rewrote all current eligibility policy
- A full end to end process mapping of the grant review process was created
Evolution
- Kishorditya’s Mandate Delivery Report outlining a Six Sigma style system review health metric for FDD to overcome Goodhart’s law
- Onboarding experience overview article on Mirror by JCook
- First season of source council was a success and even implemented accountability measures
- Developed FDD decision making process to encourage autonomy
- FDD contributor experience survey implemented
- Designed PEC model for bottoms up governance issuance within workstreams
- Created a data science pipeline for requests from other workstreams
- Created a data repository for public gitcoin datasets
- Designed architecture for GitcoinDAO data storage layer
- Made improvements/refactors/optimizations/bug fixes in SAD codebase
Season 14 Goals
FDD has Top Level Goals which seasonally examine its respective medium term focus tagged Essential Intent. Consequently, (round over round) RoR Health Metrics are deployed to determine progress toward the medium term (12-18 months) goals.
Objectives & Key Results (OKRs) are used to guide each top level goal set within a season.
Outcomes are derived from our retro & strategy workshops, discussed with CSDO leadership, then assigned to an Outcome Owner through a process of nomination, synthesis, and ratification. Outcome owners are allowed to participate on the FDD Source Council. This gives them voice through the progressively decentralized FDD decision-making process. The corresponding Decision Making Agreement can be found in Notion.
Most importantly, Outcome Owners are accountable for successfully executing on deliverables during a season and transparently communicating blockers and impediments preventing or delaying the delivery of key results. The FDD Source Council is responsible for incubating ideas and implementing passed proposals in the interest of the continuous improvement and progressive decentralization of FDD & Gitcoin.
Essential Intent & RoR Health Metrics
These metrics are how we show the round over round improvements of the FDD workstream.
Sybil Defenders
To minimize the effect of sybil attacks on Gitcoin Grants now and in the future
- Cost/evaluation (S12=$4.39;S13=$1.42)
- Inter-reviewer reliability score (n/a)
- USD removed from fraudulent allocation (S11=$606k;S12=$621k;S13=)
- Number of models (S11=2;S12=2;S13=2)
- Human evaluations (S11=1,800;S12=6,000;S13=12,000)
Grant Intelligence Agency
To improve the legitimacy, scalability, and quality of grant eligibility processes
- Grant time to activation (S11<96hr;S12=<48hr;S13=<48hr)
- Cost per review (S13=$5.38)
- Total # of reviewers (S11=7;S12=8;S13=7)
- Poison pill score (Quality) (n/a)
- Successful appeals/total new applications (S13=16/33=48%)
Evolution
To empower contributors to help FDD deliver on outcomes and become sustainable
- Contributor UX score
- Time from proposal to pass/fail
- Grant round execution score
- GIA Execution Score
- Sybil Execution Score
The evolution scores will be baselined during season 14.
Season 14 Top Level OKRs
These seasonal objectives and key results highlight our focus within Season 14.
Sybil Defenders
Objective: Improve the quality of data inputs & processes in preparation for Grants 2.0
- KR1: Add 2 anti-fraud components with Grants 2.0 applicability
- KR2: Lower the cost per evaluation by 10% while maintaining inter-reviewer reliability
- KR3: Add the community model to the “meta-model” for sybil detection
Grant Intelligence Agency
Objective: Collaborate to improve scalability and quality of reviews in preparation for Grants 2.0
- KR1: To re-review the entire set of active grants in prep for Grants 2.0
- KR2: To reduce the number of outstanding appeals and disputes to zero
- KR3: Lower the cost per grant review by 50% while maintaining quality
Evolution
Objective: Build the workstream to be sustainable and remove “key person” issues
- KR1: A contributor UX score improvement of 10%
- KR2: Find and plan 5 opportunities for revenue or protocol funding
- KR3: Build metric dashboards for each top level goal set
Season 14 Budget Requirements
Current Holdings
(Using spot prices of $5.42 GTC & $2,875 ETH)
FDD Primary Multisig (This entire wallet is counted as reserves on hand at the end of S13)
- GTC = 101,115 = $548k
- USDC = 87,585 = $87k
- ETH = 17.18 = 49k
- DAI = 6,000 = $6k
- TOTAL = $690k
FDD Ops Multisig (approximately $107k will be used for S13 epoch 11,12,13 payments)
- GTC = 16,876 = $91k
- USDC = 15,616 = $15k
- ETH = 0.375 = $1.5k
- TOTAL = $107k
FDD Payments
- GTC = 0
- USDC = 0
- ETH = .3
- DAI = 0
Team Updates
Source Council (Core contributors) - 9
FDD Roles
- Collaboration
- Talent
- Accounting
- WS Lead
- CSDO 1,2,3
Full (Trusted) Contributors (Not SC)- 18
- Multisig Keyholders
- Subject Matter Experts
- FTE
- PTE
Human Intelligence Task(HIT) Roles
- Grant reviewers - 8
- Human Evaluators - 37