[S15 Proposal - DRAFT] FDD Budget Request

This is an initial funding proposal for the Fraud Detection & Defense (FDD) Workstream requesting funding for Season 15 (1 August 2022 through 31 October 2022).

Essential Intents

In S15, FDD has components of our work that focus on all four of the DAO’s Essential Intents - though our priorities will be on the following EIs in the given order:

  • Grant Program Success
  • Protocol Adoption
  • DAO Organization & Financial Sustainability

More Information on the Essential Intents can be found in this post.

TL;DR

In Season 15, FDD aims to provide the Gitcoin community with the knowledge that matching funds are allocated as optimally as possible within the given set of assumptions and constraints. One focus will be operationally executing the fraud detection & defense needs of the Gitcoin main/Ethereum round & any other community moderated rounds. The second is assisting Gitcoin Product Collective with operational insights, quantitative analysis, and algorithmic/mechanism design research.

Based on what we learned in Season 14, we will improve our communication during the season to remove confusion regarding FDD priorities, purpose, and alignment with GitcoinDAO.

:bulb:For more context regarding our vision, the current state of FDD and more please refer to the :page_facing_up:supplement document.

Amount

FDD is requesting $349,500. This budget does not include the additional 60 days of reserves. Breakdown of the budget per budget category and initiative can be found at the end.

Gitcoin Season Season 15 Season 14 Season 13
FDD Season Budget $349,500 $330,000 $596,295
FDD Season Reserves $233,000 $220,000 $596,295
Unspent Reserves % 1) 100% 100% 100%
Treasury Request 2) $362,500 $281,041 $879,590

1) The amount of GTC requested and the value of the reserves will be adjusted based on the current market value at the time this proposal is moved to Tally using the lower of the current price or the 20 day moving average, whichever is lower.

2) The Season 15 treasury request amount is an estimate using the price of $2.99 GTC to calculate the total of the reserves rolled over from Season 14.

Milestone Report

Legend
:green_circle: Success
:yellow_circle: Incomplete, will hit goal or priority change
:red_circle: Incomplete, will not hit goal
:black_circle: Canceled - out of workstream’s control

Initiative Objectives Past Season
Value Delivered
Grant Eligibility

Executes the critical functions associated with grant eligibility reviews, disputes, and appeals.
:green_circle: Reviews and approve/deny new grant applications in under 48 hours

:green_circle: Ensures all disputes and appeals are heard and executed

:green_circle: In depth review of suspect grants to ensure minimal improper matching allocation

:green_circle: Ensure a proper functioning integration from Grants 1.0 backend to Ethelo

:green_circle: Provide data to be used in evaluating reviewer decision making issues at scale
Over 3,000 evaluations at a 39% lower cost of review

Over $80k reallocated/squelched based on suspected Sybil/Fraud investigations

Average time for appeals was brought below 96 hours allowing for in-round results

We used Ethelo to acquire over 400 reviews of the main round disputes to compare to our results with 3 “classes” of reviewers including anyone, DAO Contributors, and trusted reviewers
Data Operations

Data center which handles access control, ETL, and process automation.
:green_circle: Runs the end-to-end process used in production to detect sybil accounts

:green_circle: Supports other FDD dev ops needs & liason with Gitcoin DevOps team

:green_circle: Executes post round data analysis for grants round governance brief

:green_circle: Identifies continually evolving sybil behavior to be codified in the detection system

:green_circle: Statistically validates the algorithm and directly squelching sybil users

:green_circle: Continually trains the ml algorithm to think like the community, not the engineers

:yellow_circle: Builds containers and automations to make processes consistent and easy to learn
Identified multiple new signals

It was run every 24 hours throughout the round keeping estimations more in line with final results to improve UX

Supported Ethelo integration, Community Intelligence, and helped Grants Ops round partner experiment with Ethelo (Not advised by FDD, but supported with DevOps help)

GR 14 Governance Brief
Community Intelligence

Data analysis, model prototyping & validation, and sybil strategy center.
:green_circle: Data analysis to find the best models for passport sybil defense

:green_circle: Identifies new opportunities (low hanging fruit} for additional data-driven sybil detection strategies which leverage the data already collected

:green_circle: Fulfills opportunities & requests for further collection and organizing of data

:black_circle: Prototyping new features and models before pushing them into production
Kyle/Joe agreement to put community model on hold

:black_circle: Open source feature engineering pipeline turns behavior patterns into
passport stamps
Kyle/Joe agreement to put community model on hold
Analysis of onchain behavior

Done. Presentation for GPC on 8/3

Identified 3 new models to lower human evaluator time to insight

Designed contributor DNA

Provided recommendations to GPC Passport team for high signal stamp opportunities

Ran data analysis for GR14 contribution insights across rounds
FDD OS

Core operations & communications supporting FDD squads and contributors.
:green_circle: Funds and directs the roles requested by DAOops

:green_circle: DAO facing collaboration point to help communicate FDD role in the DAO

:green_circle: Building on last season deep dive understanding of Is FDD successful in it’s mandate

:green_circle: Facilitates and synthesizes bottoms-up and top-down input to draft FDD outcomes

:green_circle: Ensures that contributors are able to focus on the primary problems to be solved

:green_circle: Creates accountability for outcome owners via strategic cadence and collaborative decision making processes which are seen as legitimate

:yellow_circle: Building dashboards for stewards & FDD with FDD round over round metrics
Put on the backburner due to delay of Kyle/Joe alignment.

:black_circle: Removes key person issues from the workstream:
Not realistic this season with the even/overs from CSDO
Roles were implemented and executed peer reviews, compensation evaluation, and onboarding training.

FDD Review is a series of posts started to provide honest critique

WS Retros, Outcome Discovery, and Budget Discovery brought bottoms-up ideas together with GitcoinOS budget planning.

Async working agreement lowered the required meeting hours while Work-in-Public sessions gained voluntary participation helping to spread context and share methodologies.

A decision making process agreement was passed and used.
Passport Truthseekers

Pure research finding insights & sybil stories.
:black_circle: Creates patterns to be engineered for Passport to allow for specific GitcoinDAO uses

:black_circle: Provides model for grant review stamps to be sent to user passports:
Kyle/Joe agreement to hold this budget until fully aligned with GPC. Ended up not starting this squad. Moved work to Community Intelligence and Data Operations.

S15 Objectives

Based on the streamlining and aligning efforts of our work preceding S14, we consolidated our initiatives once more and will focus on 5 key objectives for Season 15.

In addition we are outlining our Always-on Commitments as well as Shared S15 Commitments with other workstreams and/or teams of GitcoinDAO in the :page_facing_up:supplement doc.

List of S15 Objectives

Initiative
(Essential Intent)
S15 Objective Metrics Key Results
Likely Deliverables / Projects
Grants Eligibility
(EI: Grants Program Success)
The Gitcoin community can reliably expect that the platform is free of fraudulent grants and that main round eligibility approvals, disputes and appeals are consistently and fairly executed in a reasonable time frame. * Average Time to Review < 48 hrs
* Average Time to Appeal Resolution < 96 hours
* Benchmark Appeal Success Rate
* Amount Saved by Investigations
* Output an FDD eligibility score for every new grant application
- Gather data for scoring experimentation
- Partner with Grants ops to improve flow
- Create a a transparent and priortized backlog of observations for GPC
Contributor Fraud

(EI: Grants Program Success)
The Gitcoin community will have confidence that matching funds are not misallocated because of sybil accounts during GR15.

* Provide understanding of sybil impact
** Fraud Tax Estimate
** Fraud Tax without Trust Bonus
** Fraud Tax without Squelching
* Average Passport Score of Sybil Acct
* Estimated Sybil Incidence / Flagging Sybil Incidence
* Inter-reviewer Reliability
- Add heuristic escape hatches to SAD algorithm
- Improve the accuracy of the SAD algorithm
- Create framework of reports which can expedite behavior classification
- Execution of always on functions
Data Empowerment

(EI: Protocol Adoption)
FDD will empower the community by enabling the crowdsourcing of raw, anonymized data and approaches to data analysis, increasing the insight discovery rate, and resulting in a plurality of Sybil detection mechanisms
* Open API Endpoint
* Useful Insights from the Community
* Data Coverage: available / existing data
* Data availability: failed/total requests
* Ease of access and usage
* Data Contest/Hackathon Participation
* Number of useful data sets
* Number of users requesting data


- Finish cloud setup including querying & processing for ETL pipelines, ML ops flow, and a well documented data repo
- Create a plan to improve the community access to data without being a service provider
- Build a front end for participatory data science
- Build an API for dashboarding and automated reporting
- Host a post grant round data hackathon rewarding analysis deemed useful by judges across other workstreams & FDD
Protocol Research

(EI: Protocol Adoption)
Insights from processes, analysis, algorithmic research, & mechanism design research provided by FDD will aid in the acceleration of development and adoption of Grant & identity protocols. * Passport Stamp Recommendations Approved by GPC Passport
* Trust Bonus to SAD Gap
* Number of Valuable Insights Generated for GPC/MC
* Key Partner Satisfaction Score
** Passport - Leon
** Grant Hub - Michelle
** Identity Staking - Brent
- Passport Collaboration: Recommendations for high signal stamps, a scoring algorithm for S15, and a post round analysis
- Identity Staking Collaboration: Specifying collaboration now with Moonshot team
- Cost of Forgery Discovery: Design a scoring algorithm which can provide sybil guarantees up to a $ amount of goods received and supporting math proofs
- Grant Hub/Round Manager Collaboration: Provide at minimum 3 scoring outputs from FDD grant reviews
- Review Protocol: Identify incentive mechanisms for sustainable sybil hunting
FDD OS

(EI: DAO Organization, Financial Sustainability)
FDD will seamlessly interface with Gitcoin OS in a way that increases cross workstream collaboration, executes on deliverables, and provides a rewarding contributor experience. * Execution Health: # of healthy OKRs
* Compensation/Happiness Ratio
* Initiative Velocity: Task/Initiative by Weeks Passed
- Build FDD Contributor Dashboard with metrics
- Build FDD House of Tasks to better track progress
- Establish mid-season OKR health check-ins
- Regulary post new articles on Gitcoin’s Gov Forum
- Have a seasonal, vibey, shared and celebrational ritual to help facilitate L.E.A.R.Ning from our mistakes
- Run CX Survey / Satisfaction Review
- Conduct a facilitated process with Stewards to guide FDD priorities
- Create visualized product/FDD roadmap for upcoming seasons




Budget Breakdown

View: USD per Initiative

Initiative Amount USD %
Grants Eligibility
(Grants Program Success)
$63,000 18.0%
Contributor Fraud
(Grants Program Success)
$84,250 24.1%
Data Empowerment
(Protocol Adoption)
$80,000 22.9%
Protocol Research
(Protocol Adoption)
$54,250 15.5%
FDD OS
(DAO Organization, Financial Sustainability)
$68,000 19.5%
Total $349,500 100.0%

View: USD per Category

Budget Category Description Amount USD
Core Contributors 2 WS Leads
Product Manager (Joe)
Project Manager (Tigress)

4 Initiative Leads
Grants Eligibility (Zer8)
Contributor Fraud (SirL)
Protocol Research (Kish)
Data Empowerment (Zen)

1 Technical Lead
Data Science (Omni)

1 Data Analyst (Adebola)
$238,500
Trusted Contributors Technial Writer & Science SME (J-Cook)
DeSoc SME (Carl)
Data Science SME (Takuya)
Analyst (Sorana)
SWE (Eric & Yogeesh)
$52,000
Hiring, Bounties, HIT Roles Data engineer New Hire - $25k
Bounties & SaaS - $7,5k
Human Evaluations (SAD) - $5k
Grant Reviews - $15k
Accounting - $6,5k
$59,000
Total $349,500
5 Likes


Visualization of the budget draft above. I intend to post these under each budget and keep them updated as proposals incorporate feedback and changes. Please reach out if you have any questions or input.

3 Likes

The addition of the infographic is a very cool idea. Nice job Fred :slight_smile:

3 Likes

This budget is by far the best (i.e. most easily understood) budget I’ve seen and I’ve been looking for a long time. Huge thanks to everyone involved.

The info-graphic is amazing. Thanks so much for that.

Request: Can you include the number of full and part time contributors on the info-graphic as this would allow me to do a quick estimate of per hour rates, which, while perhaps somewhat distasteful, are undeniably useful from a “reasonableness” perspective.

Those numbers appear in the budget doc, but can they be added to the info-graphic? Is the list of contributors at the bottom complete (there are seven)? Is that indicative of the number of contributors?

4 Likes

Thanks for the feedback regarding the visual. Every S15 budget proposal follow a structure that takes care of the broader strokes, but there are still a couple of differences when you really dig into the details. FDD for example does not present a list of full time and part time contributors so that information can’t be added to this visual at this time. Also a bit tricky as WS use the term “core contributors” differently. I hope we can get some unifying rules in place for the non-draft budget proposals.

The recent changes in workstream composition resulted in quite a bit of changes, somehow Zen didn’t make it in my edit earlier, he/she has now been added.

3 Likes

For the record, we have been pushing to get a definition of the term core contributors at CSDO. How do you see the other streams defining it differently?

Currently, the 8 core contributors listed can be compared to a full time salary.

The trusted contributors are hourly ranging from $20/hr to $75 hr.

Thanks! I can confirm this is an accurate update of the 8 core contributors for FDD.

5 Likes

At the risk of being repetitive this season - what a great write-up! Thank you for this clear write-up and for all the work you do to secure the credibility of Gitcoin et al

Just a couple of quick ones:

  • where does cloud or data center spending show up?
  • do we anticipate it growing as we gather more data?
  • would FDD potentially generate revenues in the future as other platforms embrace Grants 2.0 and need “our” / your protective services?

What have been the most difficult areas of responsibility to clarify between FDD and other workstreams?

3 Likes

Thank you @epowell101 for your feedback.

Here are our answers to your questions:

Question :one:

Spendings for software as a service does show up in the budget breakdown. However, the visibility of SaaS expenses could have been outlined more clearly.

SaaS expenses for cloud / data center are considered to be part of the initiative Data Empowerment in our percentage split estimates. The current cost is < $250/month for Google Cloud Platform & Github.

At some point we might look into and explore decentralized cloud solutions.

Question :two:

Yes, but not a significantly (< $1,000 monthly) anytime in the next 3-6 months. Much of what we are doing is making

  • great documentation for which subgraphs and open web3 APIs are best
  • keeping current data sets clean for crowdsourced analysis.

Neither FDD nor our initiative Data Empowerment is intended to be a data services organization.

Question :three:

This is a spicy :hot_pepper:question.

Our current answer is: We highlight that FDD is aiming to protocolize solutions, not to be a services company of any sort. That said, there are definitely avenues for monetization including spinning out a service DAO or even building and owning digital public infrastructure for sharing data, feature and model engineering for ML, and crowdsourcing data science insights.

However, one could argue and ask: why does protocolizing seem to be at odds with generating revenue? Which one applies:

  1. Revenue XOR Protocolizing
  2. Revenue OR Protocolizing

The need is for buy in from the rest of the DAO and stewards to give us the freedom to discover where the value is.

To make that happen, the stewards community would have to reconsider their decision, because they voted DOWN a discovery for a revenue-generating sybil detection DAO. We understood that the driver behind this decision was continually viewing attempts of revenue generation as “defecting”.

Question :four:

@DisruptionJoe has been leading most of alignment conversions and he outlines:

One area is our work toward solutions which decentralize the moderation inputs to a grant round. User and grant moderation is one of our core competencies. Our bottoms up suggestions all recommend that we build digital public infrastructure to sustainably serve these needs in a credibly neutral way that is equally accessible to all the communities that will run their own rounds in the future.

The stewards, guided by a few strong opinions, however, have pushed us to centralize our efforts for efficiency and simply execute until a time when the other workstreams can design protocol-based solutions. One of our strengths is Protocol Research. We believe our research could be much better utilized to align roadmaps and context awareness across the DAO.

Here are some hands-on examples:
Example 1 - Using Ethelo for scaling grant reviews
Example 2 - Discovery for Sybil Detection DAO
Example 3 - Shutting down the community model only to be wowed by a presentation of on-chain features.

And …

The best example - We couldn’t get our contributors’ Data Access to cGrants database until last season for all but Blockscience. We couldn’t tinker with data to improve rounds over rounds.

So then we literally moved the whole SAD model to be run by the DAO with no way to get or check data other than requesting it from Blockscience and scraping it. (We didn’t have any access to data prior to season 14.)

:raised_hands: We like to highlight that alignment efforts made by other workstreams, especially GPC, since last budget season have made drastic improvements for us in all the mentioned areas. We do feel the results of better communication about these issues. :blue_heart:

Hope this helps :pray:

2 Likes

I seem to have stepped in it a bit w/ my :hot_pepper: question.

I’m a big believer in a sure sign of a good org being some people are “disagreeing and committing.” That’s the sense I got from your post which speaks highly IMO both of FDD’s quality and that the overall DAO is making tough decisions (setting aside whether or not they are correct :)).

That said, as someone w/ too much experience in the old open source world & starting stuff & now DAO metagovernance largely via Wildfire - I’m super curious about the 4 listed areas and overall the notion of relevant digital public infrastructure. If I could be useful as a volunteer in listening, helping to further crystalize, and so on please count me in. I’ll ping you on the channels as well.

BTW thanks all for the NFT.