[Cancelled/Superceded S14 Proposal] FDD Season 14 Budget Request

đź“ś Proposals
1

Newest proposal: https://gov.gitcoin.co/t/proposal-fdd-season-14-budget-request/10658

Workstream Mandate: Defend Gitcoin (Grants, DAO, & future products) from threats to its legitimacy, credible neutrality, and sustainability.

Season 14 Budget Proposal

This is a renewal proposal for the Fraud Detection & Defense workstream requesting budgetary funds for Season 14 (1 May 2022 through 31 July 2022).

FDD is requesting $486,130 to fund Season 14 (S14) and replenish its reserves for S15.

High Level Budget Breakdown

Season 14 Budget Total = $649,455

PLUS Season 15 Reserves = $432,970

Total Needed = $1,082,425

MINUS Current Reserves = $596,295

($690k total holdings | $596k Season 13 reserves & $94k treasury)

Total Amount requested at this time = $486,130*

*The GTC total will be adjusted based on the current market value at the time this proposal is moved to Snapshot and to Tally. The vote will be to send $ in GTC at the current price when moved to Tally or the 20 day moving average (whichever is lower).

TLDR

FDD is responsible for the following items which are mapped to the outcome sets below

  • Defending Gitcoin Grants from exploits. (Sybil Accounts, Collusion, etc)
  • Coordinating the grant eligibility policy & reviews
  • Sustainably operating the workstream. Goal setting, delivery, accountability, & comms.

Sybil Defenders (Legitimacy)

  • Maintains DevOps for Sybil Detection machine learning pipeline
  • Designing models for decentralizing the inputs to the machine learning pipeline
  • Humans evaluating accounts to determine if they are sybil and train machine learning
  • Solving for sustainable mechanism design minimizing the exploit surface area/profitability
  • Inform decisions and improve our resilience to fraud using research, simulations, NLP/ML, etc

Grants Intelligence Agency (Credible Neutrality)

  • Set up tooling to lower our cost, improve the quality of reviews, and give transparency/consistency to the decision processes
  • Execute all grant reviews, disputes, and appeals to ensure grants rounds run fairly
  • Design, test, and implement a model for decentralizing grant reviews using incentives
  • Deep dive reviews of grants to better understand the methods attackers use

Evolution (Sustainability)

  • Ensuring governance decisions and processes are properly designed and implemented
  • Supporting payments and advancement of smart contract usage in comp models
  • Articulating the work being done in FDD for internal and external stakeholders
  • Setting strategy goals and ensuring that the workstream is accountable for delivering
  • Providing data analysis services for FDD (this season we are opening to all GitcoinDAO)

Season 13 Recap

FDD Season 13 Report Card

GR13 Governance Brief

Key highlights & learnings:

Sybil Defenders

  • GitcoinDAO now runs the full anti-sybil operational process (previously done by Blockscience)
  • Doubled the number of human evaluations done at â…“ of the cost
  • Classified sybil behaviors & ran impact analysis
  • Performed trust bonus analysis and provided recommendations
  • Won Metapod pitch competition and validated users for Sybil Detection DAO
  • Reviewed sybil assumptions and began work on mechanism design & simulations

Grants Intelligence Agency

  • Hit all primary round OKRs without getting data integration on time from Holdings
  • Assisted PGF ops in tag requests for eco & cause rounds
  • Began creation of “trusted seed” of reviewers
  • Designed curation incentivization structure for any community to have community curation
  • Grants Curation NLP model that can replicate the GIA team’s decisions with 69% accuracy
  • Reviewed and rewrote all current eligibility policy
  • A full end to end process mapping of the grant review process was created

Evolution

  • Kishorditya’s Mandate Delivery Report outlining a Six Sigma style system review health metric for FDD to overcome Goodhart’s law
  • Onboarding experience overview article on Mirror by JCook
  • First season of source council was a success and even implemented accountability measures
  • Developed FDD decision making process to encourage autonomy
  • FDD contributor experience survey implemented
  • Designed PEC model for bottoms up governance issuance within workstreams
  • Created a data science pipeline for requests from other workstreams
  • Created a data repository for public gitcoin datasets
  • Designed architecture for GitcoinDAO data storage layer
  • Made improvements/refactors/optimizations/bug fixes in SAD codebase

Season 14 Goals

FDD Season 14 Goal Planning

FDD has Top Level Goals which seasonally examine its respective medium term focus tagged Essential Intent. Consequently, (round over round) RoR Health Metrics are deployed to determine progress toward the medium term (12-18 months) goals.

Objectives & Key Results (OKRs) are used to guide each top level goal set within a season.

Outcomes are derived from our retro & strategy workshops, discussed with CSDO leadership, then assigned to an Outcome Owner through a process of nomination, synthesis, and ratification. Outcome owners are allowed to participate on the FDD Source Council. This gives them voice through the progressively decentralized FDD decision-making process. The corresponding Decision Making Agreement can be found in Notion.

Most importantly, Outcome Owners are accountable for successfully executing on deliverables during a season and transparently communicating blockers and impediments preventing or delaying the delivery of key results. The FDD Source Council is responsible for incubating ideas and implementing passed proposals in the interest of the continuous improvement and progressive decentralization of FDD & Gitcoin.

Essential Intent & RoR Health Metrics

These metrics are how we show the round over round improvements of the FDD workstream.

Sybil Defenders

To minimize the effect of sybil attacks on Gitcoin Grants now and in the future

  • Cost/evaluation (S12=$4.39;S13=$1.42)
  • Inter-reviewer reliability score (n/a)
  • USD removed from fraudulent allocation (S11=$606k;S12=$621k;S13=)
  • Number of models (S11=2;S12=2;S13=2)
  • Human evaluations (S11=1,800;S12=6,000;S13=12,000)

Grant Intelligence Agency

To improve the legitimacy, scalability, and quality of grant eligibility processes

  • Grant time to activation (S11<96hr;S12=<48hr;S13=<48hr)
  • Cost per review (S13=$5.38)
  • Total # of reviewers (S11=7;S12=8;S13=7)
  • Poison pill score (Quality) (n/a)
  • Successful appeals/total new applications (S13=16/33=48%)

Evolution

To empower contributors to help FDD deliver on outcomes and become sustainable

  • Contributor UX score
  • Time from proposal to pass/fail
  • Grant round execution score
  • GIA Execution Score
  • Sybil Execution Score

The evolution scores will be baselined during season 14.

Season 14 Top Level OKRs

These seasonal objectives and key results highlight our focus within Season 14.

Sybil Defenders

Objective: Improve the quality of data inputs & processes in preparation for Grants 2.0

  • KR1: Add 2 anti-fraud components with Grants 2.0 applicability
  • KR2: Lower the cost per evaluation by 10% while maintaining inter-reviewer reliability
  • KR3: Add the community model to the “meta-model” for sybil detection

Grant Intelligence Agency

Objective: Collaborate to improve scalability and quality of reviews in preparation for Grants 2.0

  • KR1: To re-review the entire set of active grants in prep for Grants 2.0
  • KR2: To reduce the number of outstanding appeals and disputes to zero
  • KR3: Lower the cost per grant review by 50% while maintaining quality

Evolution

Objective: Build the workstream to be sustainable and remove “key person” issues

  • KR1: A contributor UX score improvement of 10%
  • KR2: Find and plan 5 opportunities for revenue or protocol funding
  • KR3: Build metric dashboards for each top level goal set

Season 14 Budget Requirements

1259Ă—363 59 KB

1265Ă—839 134 KB

Current Holdings

(Using spot prices of $5.42 GTC & $2,875 ETH)

FDD Primary Multisig (This entire wallet is counted as reserves on hand at the end of S13)

  • GTC = 101,115 = $548k
  • USDC = 87,585 = $87k
  • ETH = 17.18 = 49k
  • DAI = 6,000 = $6k
  • TOTAL = $690k

FDD Ops Multisig (approximately $107k will be used for S13 epoch 11,12,13 payments)

  • GTC = 16,876 = $91k
  • USDC = 15,616 = $15k
  • ETH = 0.375 = $1.5k
  • TOTAL = $107k

FDD Payments

  • GTC = 0
  • USDC = 0
  • ETH = .3
  • DAI = 0

Team Updates

Source Council (Core contributors) - 9

FDD Roles

  • Collaboration
  • Talent
  • Accounting
  • WS Lead
  • CSDO 1,2,3

Full (Trusted) Contributors (Not SC)- 18

  • Multisig Keyholders
  • Subject Matter Experts
  • FTE
  • PTE

Human Intelligence Task(HIT) Roles

  • Grant reviewers - 8
  • Human Evaluators - 37
15 Likes
2

Love to see the evolution of FDD ws, this is a very clear budget proposal including budget request and reserve and last season financial report.

The most inspriing part is the structures and the system. OKRs driven diffirent teams and clear responsibilities, and final result is the cost is decreasing, and the sybil is more strong.

Definitely say “YES” for this budget request.

11 Likes
3

this season we’re putting an emphasis on higher inter squad cohesion and minimizing the duplication of work between squads by consolidating!

season 14 for FDD is gearing up to be the most exciting and impactful one yet! our new members (myself included) have really started hitting our strides and finding where we belong.

shout out to @DisruptionJoe for putting all this together and cutting out as much noise in our governance/budgetary decisions as possible.

11 Likes
4

Since I oficially joined Gitcoin DAO(from GR11) I can say that it’s the most impressive part for me has been the ever evolving structure of the FDD and the way it’s descentralizing while maitaining its purpose to defend the DAO against threats to its legitimacy, credible neutrality while also ensuring scalability. I can say that @DisruptionJoe has helped everyone in the FDD evolve and learn + he ensured that the FDD can and will help every workstream, he’s a true leader, a very deep/thoughtful one and also an operational genius. I’m lucky to work with him, with my collegues in the FDD and also with the GItcoin DAO peeps

I with to invite anyone interested in voting on our budget to study the FDD’s evolution from GR11 to GR12 to GR13 and now to GR14. It sound be eazy to follow up by reading our governance briefs from the respective seasons.

The cohesion of this workstream is amazing and quality, values and proactiveness of it’s members are something rare, even in web3.

Evolving while also maintaining and improving the FDD’s initial functions is not an easy task and should be taken into account when voting on this budget request.

As a SC member and driver of 2 squads in the Grant Investigation Agency I believe it’s crucial we continue to get funded. We have been responsible with overviewing and executing on one of the most complex and important matters for the Gitcoin: the Gitcoin Grants!

Congratulations for all the FDDers, you are great!

10 Likes
5

Really excited to improve the Human Evaluation and the app build with omnianalytics, data quality and reviews, continuous training to detect Sybils Attacks and lower the Fraud proportion to redirect these amounts in the right direction.

Looking back since GR12 we were able to refine the overall development of the Fraud & Detection workstream but not limited to and advance our sharable values/goals toward a common mission.
We still have plenty of work that need to be done but this is clearly a step forward.

Shoutout to @DisruptionJoe @David_Dyor @tigress @ZER8 @nollied @vogue20033 @bobjiang @wkarshat @kishoraditya basically the hole FDD Team and all new comers that joined lately like ZEN who did work with me on the ASOP and of course @danlessa with his team BlockScience!

10 Likes
6

Season 14 in the FDD will herald more inter squad solidarity, which will increase the success impact by the workstream. The bottom up structure of the FDD makes it easy to come up with feasible and sustainable ideas, while still maintaining the “community first” promise of a DAO. The growth and evolution of FDD when i joined in GR12 has been unprecedented, hence an indication that S14 will be impactful and exciting.
Kudos to @DisruptionJoe for shaping out a cutting edge structure for the workstream.

10 Likes
7

Since I have joined the FDD Workstream in January, I have met the brightest and most passionate and innovative people. I’m so proud to be a contributor to FDD and GitcoinDAO. And thanks to the support and encouragement of my FDD family I am so excited and motivated to contribute even more and collaborate with all the contributors at FDD and GitcoinDAO.
Thank you everyone :blue_heart:

9 Likes
8

we also all love to argue, a healthy sign of any community :slight_smile:

@ZER8 @Sirlupinwatson @tigress @DisruptionJoe @octopus and others :heart:

8 Likes
9

I admire FDD’s commitment to decentralization and you inspire other workstreams to do better.
As a workstream FDD’s tools to measure their effectiveness and impact makes this workstream easy to trust, and I can’t wait to see how FDD evolves. :handshake:

8 Likes
10

S14 is going to see some spectacular results with FDD gearing towards grants 2.0, Sybil DAO, and understanding deeper insights to be the backbone of Decentralized Autonomous Organizations in general!

Awesome work putting down the 90-day vision and budget!

7 Likes
11

Another clear well-described budget from Disruption Joe that gets my full support. Glad to be part of the team with you and the entire FDD & Gitcoin Dao. I appreciate the leaning out and trimming of the budget plus getting some metrics established to help communicate.

6 Likes
12

Lol. We never argue. I can argue that we don’t ever argue :sweat_smile:

3 Likes
13

@DisruptionJoe is a master at building the machine that builds the machine. FDD has prepared a lot of content but its buried in several documents, I’d suggest maybe making it a bit easier for people to understand how FDD is functioning.

Here’s the approach I took to make sense of the FDD budget that I took (after looking at everything) and the subsequent comments/suggestions I’ll give.
1. Get a sense of the top level sums
This can be found under Season 14 Budget Requirements and splits the budget into outcome owners, GIA, Sybil Defenders, and Evolution
2. Start with the FDD Season 14 Goal Planning documents
3. See if you agree with the overall goals

Here are my thoughts for the team:

  • What about prevention of sybils?: I’m curious if FDD is looking at prevention/deterrence rather than detection. having less sybils might make it easier to defend too. A red team approach might help where some FDD individuals are empowered to pose as sybils to try to attack us and learn from that rather than relying too much on automation and AI/ML.
  • FDD Storytellers might not be necessary: this stood out as its more of a PR activity for FDD. For context, its asking for 14.4k USD which is >50% of most of your sub-workstreams. Its unclear how it will help improve contributor UX, GIA execution, or sybil execution and the items of quests etc seem to fall under onboarding instead. I’ll suggest removing it or making it into just a sample blog post or explainer for contributors.
  • Greater focus on policy: working with grantees for GR13, there was some feedback that the policy was unclear or that the process for grantees might feel arbitrary or disheartening. I know this isn’t the case but perhaps reallocating the budget from storytelling to policy communications will be more helpful. In fact, including that as a top-level OKR for GIA might be even better. Right now Evolution has a higher % of budget compared to GIA too which might suggest overindexing on automating and growing FDD for this particular budget
  • Inter-Workstream work: would like to see some evidence of discussion on FDD with Grants 2.0 especially with “Catalyst | nollied” which is asking for 37.5k USD (one of the largest allocations). Caveat that I’m not saying there isn’t discussion, but would like some idea of how aligned this work is especially because what is proposed is very experimental and is researching a lot of different highly technical topics.
  • Sybil Detection DAO needs more details: I know there’s some context as during ETHDenver this idea was floated. Right now I’m assuming its about creating a sub-DAO focused on sybil detection, but I thought that was FDD itself. Also it’s asking for 90k USD which is the largest % of the overall budget.
  • Overall thoughts: I like that FDD takes time to explore more experimental issues and DAO-wide issues. The budget and its process to come up with it is democratic and decentralised which I appreciate too, however I’d like to see work done between other parts of the DAO (links to discussions, WIP, etc) to get a better idea how the work is going.

4. Take a look at past performance with the S13 Report Card

  • at first glance, most of everything has been done, so congratulations to the workstream for pulling of a great season!
  • interesting to see the work exploiting grants and a focus on simulations, I suppose that falls under prevention!
  • curious how some of these things have been used like the mandate delivery option and the CAP table research

5. Requests and rationale for vote choice

  • Requests: Additional focus on Grant Policy communication, removing FDD Storytelling, include more inter-WS discussions to provide more confidence on decisions and direction. I’d like to see more work from subject matter experts on the technical aspects of FDD, especially on mechanism design and AI/ML work
  • Rationale: I am leaning towards a FOR vote as I agree with most of the OKRs and won’t want to let one OKR hold back my general positivity towards FDD.

Once again a great job putting together elements of the budget!

5 Likes
14

yes!!! that’s what my squad “Catalyst” is researching and developing. I’m a firm believer that we should not totally rely on retrospective sybil account detection. we’re working to minimize the profit margins for exploits in general using novel mechanism design proven through simulation. i’m hoping to coordinate with the grants 2.0 folks (ie. @kevin.olsen) to bring this to fruition.

to be clear, we haven’t had full on discussions yet, but I am hoping that this season we can work more closely. either via direct communication, or by presenting our propositions and simulation results to be implemented into the “marketplace” of components/presets for the round managers. the only “evidence” i have can be found here.

3 Likes
15

storytelling is something missing in nearly all streams, esepcially FDD. there’s an almost magic-like property of taking what’s going on internally and communicating it in a digestible format. i think of it like self-awareness and external relationship building.

you know, in cognitive behavioral therapy (CBT), communicating with ones-self unlocks immense potential in becoming a better human being. why wouldn’t this apply to the shared consciousness of a DAO?

3 Likes
16

Thanks for this suggestion. We also made this budget review tool on Ethelo and would love for you all to check it out to review all of the Season 14 budget requests! https://gitcoinbudget.ethelo.net/

Yes. This is directly a goal of ours with the Catalyst squad (formerly called Matrix).

We actually found that this one bubbled up for us from all the contributors. We weren’t doing a good job of sharing context within the workstream between squads. Most of your other criticisms here seem to highlight the need for some level of comms.

This is a very small budget and the outcome owner is already a source council member with another outcome so there isn’t a full time person dedicated to this. We simply felt the need to be intentional about this.

One answer to this is that we raised the round management budget substantially and put the grant policy work within this stream. We believe that the policy errors were more process driven than due to an unclear policy (although its just a weighting, not a binary).

Another answer is that we are bearish on policy in general. Instead, we are solving for how any community, including our own, can collectively set and review criteria and grants eligibility status against that criteria in a decentralized way that maximizes for trusted outcomes first, then minimizes cost for scalability.

You can see this in our Rewards Modeling OKR report here or if you want a high level overview, watch this 8 minute video.

This workstream was formerly called Matrix. We believe that it’s work is crucial because without it we are not preparing for a grants 2.0 future. Funds removed from fraudulent allocation is the most relevant metric for them.

Without them we do not have a high-level overview of why we are focusing where we are. In the last season, Kylin’s review on the matching caps informed the grants ops decision on what to do with matching caps.

Additionally, they identified an entirely new vector of attack. Their work is needed in my opinion and their budget is only lower than the sybil model now because innovation and research is a hard sell, but unfortunately FDD is tasked with solving 2 unsolved research problems for Gitcoin.

In season 14, they are tasked with coming up with Grants 2.0 components which can minimize fraudulent allocation. They will map the composable components and communicate an understanding of the tradeoffs in different component stacks. They will also be able to inform decisions based on simulations.

Grants 2.0 is going to be a main customer for this squad because they will need a collection of components to start. They are currently focused on building the first component of Grants 2.0 as the ones which are used in cgrants, but this squad can simulate and suggest new legos with code that can be dropped into production.

They can also plugin synthetic data to the community model to help train it faster. Another way they can reduce costs is in prototyping NLP solutions. Lastly, they can help other FDD squads with simulations. Deliverables include reports and graphics and even code for grants 2.0 components.

I will be directly leading this effort. The details are in flux, but we need to begin building some of our standards into protocols which will serve as microservice protocol DAOs. The entire solution can be generalized to serve all of web 3. Here is the Sybil Detection DAO deck.

Overall, thank you for the thoughtful response.

3 Likes
17

Gotcha on that, may I suggest not asking for a budget to the tune of 5.8% of FDD’s overall budget. As this is more nascent, probably a smaller sum to start to show some results would be good + there’s a fair bit of reserves each round that can be used for activities like this

(edited for accuracy, 9.2 to 5.8%)

1 Like
18

True, storytelling is fundamental to the human experience and knowledge transfer. However that’s not the core of what FDD ought to be doing and my personal take is that focus is definitely important!
Hence the suggestion of focusing on explaining the grants policy itself and starting with that rather than creating all these experiments and quests suggested. This is something that can be measured as you can now survey past grantees and new grantees or figure a way to see if they get the policy.

2 Likes
19

The catalyst budget is 5.8% and they are only new in that it is a name change. I highlighted some results above. This squad is comprised entirely of Ph.d statisticians, mathematicians, systems engineers, and machine learning engineers. This is a very low budget for the work being done imho.

4 Likes
20

by this same logic, should we also slash accounting/talent/collaboration?

research and development is longer term for tangible results with high risk high reward. plus, we’re paying our highly qualified members below market value already.

2 Likes