[Proposal] FDD Season 14 Budget Request

Yes, these are private repo, you can request an access to @omnianalytics (OmniAnalytics#5482) on discord, just send him your GitHub username then he can add you in the group.

For the “ASOP” “SAD” @DisruptionJoe can add you in the group as well.

2 Likes

I will do that. But why private? Since we are all about opensource software here, I would expect what the DAO produces to be opensource too.

1 Like

The repos contain sensitive information, like the specific features being used for sybil detection and how they’re retrieved. Making those open source will make obvious to the attackers to what look into in order to avoid detection. I don’t see a hard blocker for opening it up, but this should be part of an involved discussion (which FDD is already doing a lot) and the precautionary principle should apply.

Also, the SAD codebase is as of now a monolithic piece of software, which makes not trivial to separate the sensitive stuff into separate repos. It’s definitely doable as a part of the FDD evolution to develop into that, and opting for increasing community involvement on Sybil Detection will definitely make that highly desirable.

4 Likes

I was talking about this season :slight_smile:

I understand, I’m actually upset because I’m sure we both have Gitcoins DAO best interest at hearth, Our solutions are not that easy to comprehend because the threats are the same.
Maybe next season there could be a Steward Budget call in which all the workstreams present their needs and accomplishments with their previous budget :slight_smile:

The simplest way of seeing what we are doing is the diagram:

2 Likes

Here is another way to view the FDD work

Grant Eligibility = Content Moderation

Round execution - Reactive operational execution (All Round Management)

  • Two reviewers per grant minimum x 1000 new grants
  • Training of new reviewers - facilitating learning and communication during the round
  • Manual documentation of flags/disputes
  • Judgements on disputes
  • Manual documentation of appeals
  • Facilitation of conversations around appeals and judgements

Short Term Improvements - Proactively improving for the next round (Mandate Delivery, Data Science)

  • Ethelo devops support to bring down per review cost - spend $10k, save $10k per round
  • Gather better data from the approvals process
  • Training
  • Research on Kleros, Celeste and staking ideas (We had this on the roadmap 2 seasons ago, but then the dissolving of dGrants was a surprise and no one included us in the product conversations around building grants 2.0 so we thought we were supposed to maintain course and figure out our own solutions for grants 2.0)

Medium Term Connecting Current Course & Speed to Future State (Trust Builders)

Long Term Vision - Future State = An ethically values aligned and sustainable solution (Trust Builders w/ Sybil Detection DAO & Passport)

  • Passport is used by everyone on earth - They have an option to participate in sybil hunting and grant curation for reward at anytime
  • The system doesn’t BAN sybils and fraudulent grant creators, it instead only allows them to play with each other
  • We avoid the web 2 moderation trap of becoming addicted to lean and inexpensive (but easily corrupted) delegated authority
  • We have built a system that offers communities the potential to choose “community curation” which is a decentralized review process aka a system where they can’t do wrong as opposed to shouldn’t do wrong
  • Stamps from this system are HIGH quality non-sybil signals

Sybil Defense = User Moderation

Round execution - Reactive operational tasks (Mostly SAD squad & Human Evals)

  • Run ASOP algorithm to identify sybil accounts
  • Push info to gitcoin backend for sanctioning/squelching (114,000 contributions out of 500k total in GR12)
  • Enough human reviews to statistically validate the model is working properly (>1,500 but decreasing returns over 8,000 or around there currently)
  • Enough human reviews to identify new behaviors of attackers (More is better as long as they are putting in genuine human subjective answers and not just using a “rubric”)
  • Enough human reviews to disperse bias across reviewer geographies, cultures, race, sex, etc.
    (rather than programming the bias of the engineers)

Short Term Improvements - Proactively improving for the next round (Data Science & Community Model)

  • Identifying the high confidence sybil users (and known not sybil) and analyzing for correlations to Passport stamps
  • Turn Passport stamp correlations into features
  • Continue work identifying sybil behavior classes and new features
  • Analyze human evaluations for inter-reviewer reliability

Medium Term Connecting Current Course & Speed to Future State (Community Model, Mandate Delivery, Data Science)

  • Remove all cgrants backend data from algos - Use only non-pii publicly available (On chain) inputs
  • Identify long term value patterns that can prove the cost of forgery
  • Continue algorithmic sybil defense for communities that need it early. Although gitcoin is building a very long term valuable solution with passport, someone still needs to read the data AND we need to compare the results of our hypothesis that dPoPP will continue to solve the problem and not be gamed. We should not lose the current working system until we have tried to falsify the new hypothesis with the best data available!

Long Term Vision - Future State = An ethical & values aligned sustainable solution (Community Model, Sybil Detection DAO, Trust Builders)

  • Sybil Detection DAO decentralized user moderation - NOT by having an ever expanding set of human evaluators, but by using the machine learnin to scale the human subjectivity. Algorithms hold unknown amounts of bias. Keeping humans in the loop is an ethical solution in line with Gitcoin’s values.
  • Dynamic reading and peer predictions
  • A high values rules based system designed and dynamically updated using crowdsourced data analysis (Community model)
  • Large ownership stake in digital public infrastructure for Gitcoin & aqueducts!!!

Note: Evo - oXS & Evo - Ops are both operational functions. The first being our decision making, meetings, calendar updates, internal comms, etc. The latter being the roles with DAO ops has requested each workstream have and our payments.

5 Likes

Another thing that might help to understand FDD is it is all OUTCOME based organization.

Sybil Defense

Are we protecting grants from sybil attacks in a legitimate way that empowers the voices of the many (Its this part about empowering voices of many where @kyle and I disagree on whether community model and human evals should continue. I agree there is a high value system being built, but it is not there yet. By putting in the evaluation work now, we can better attempt to disprove the hypothesis that the new and untested high-value system Passport creates will be ungameable)

  • This round
  • Next round
  • Making moves in line with a grants 2.0 future
  • Identifying assumptions and risks
  • Reducing system bias
  • Decentralizing the inputs where it solves a specific problem - not for the sake of “decentralization”

Grants Intelligence Agency

Does Gitcoin grants provide maximum credible neutrality in the way it moderates content? (aka grant eligibility)

  • Executing reviews, disputes, appeals for the current round
  • Developing MVPs like work with Ethelo which will scale our reviews
  • Get better data on reviews to scale even further
  • We will still have to curate the main gitcoin rounds with grants 2.0 protocol
  • Should we accept that the content moderation on web 3’s premier value network uses delegated authority a la Twitter/Facebook style unless someone else is willing to invest in a decentralized grant review protocol or should we offer the “can’t do wrong” system out of the box? If our mission is to launch grants 2.0 protocol, you might consider this scope creep. If you think the gitcoin mission is to help communities build and fund their shared needs, it is probably essential.

Evolution

Are we consistently re-evaluating and improving

  • Staying ahead of the “red team”
  • Setting metrics which aren’t subject to Goodhardts law
  • Aligned and focused internally
  • Accountable to each other
  • Have the needed context and info

I would really love very direct feedback from @lthrift (Product Director) @kevin.olsen (VP of eng) @brent (Passport PM) and @nategosselin (Grants PM) to see which specific points are not aligned here. Let’s have a productive conversation!

I am also asking for us to be involved in co-creating a future. I don’t understand why FDD was only involved in the grants 2.0 planning in discovery workshops where the product team for grants 2.0 learned what FDD was currently doing, but not in any future planning where we could have put these ideas forward and course corrected together at an earlier point in this process.

In fact, I put the proposal forward to CSDO that we needed to get on the ball, hire Sam and The Ready to discuss and share our plans before the budget proposals were due.

5 Likes

Daniel I understand that and it’s a sufficient explanation to me for having it private for now.

Try to work towards making it opensource eventually at some point soon, perhaps by separating the sensitive paremeterization out in a private repo.

1 Like

Yes. This is the plan with the community model squad. We will figure out the Passport (dPoPP) usage patterns of known sybil accounts to identify features to use in creating the Gitcoin Passport Reader score which would be exported to dapps that don’t want to roll their own algo.

Unfortunately, we only have a couple seasons to do this while the Passport is being used on cGrants before the grant registry and round manager parts of the protocol are rolled out.

The idea being we can move from private repos using some PII data to open repos using only publicly available on chain data.

We know that the community values decentralization, privacy, consent, etc. We have had the same outcomes all along and have adjusted with the new plans for grants 2.0. At this point, the community has been cheering us on as we built the proper tools for the job, but they aren’t all finished.

We thought the timeline was to be able to launch by the time grants 2.0 launches.

Now, if we are “defunded” it is a catch-22. People can say, “see they didn’t build anything with the money”, but in reality we did a lot of research and iteration to understand the problem space and have started on building in the solution space. All along we were executing the necessary minimum to keep the rounds going.

The reason I shared the documents around Trust Builders and the info about the community model and SDD above is because the evidence is tangible. If we stop now, Gitcoin community will provide a grants 2.0 protocol with all its eggs in one basket on an untested sybil defense hypothesis and throw away its investment in other solutions that work today, against the advice of those directly working on the problem, even though those people are agreeing that it is a great hypothesis that we should test!

Please discuss with the data scientists and people working on the problem before making a decision.

It would also leave a 3/4 built skyscraper behind where people could say, “see, GitcoinDAO couldn’t provide a protocol without single points of failure and they wasted all their money on this thing”… but it might just be that finishing building that skyscraper would have removed the single points of failure. (skyscraper = the sybil work done so far)

4 Likes

Thanks @DisruptionJoe for this amended budget request - I echo others’ thoughts in this being much clearer than previous iterations. I know this has been quite a process for you all and I want to acknowledge that the clarifying work is really appreciated.

I also really love the framing of a menu of budget options for stewards to choose from, rather than a binary yes/no decision. I think we should explore this in the context of other workstreams in the future.

I am not certain yet which way I will vote - I want to dive into the details a bit more - but at a first glance, I am leaning towards Option 3.

5 Likes

This proposal was updated today to reflect feedback from stewards and conversations with GPC, Passport, and Kyle at the foundation on how to better align and communicate our roadmap.

We expect to post to snapshot on Tuesday, 5/31.

6 Likes

I am reminded of the quote by the great V.I. Arnold, “Mathematics is the part of physics were experiments are cheap.”

This is one of those places where the is really about how to define the cost function that we’re trying to optimize. It’s important to think very carefully about all of the tradeoffs – not only about the budget for today, but also about the long-term values and risks in play.

I’ve seen it proposed that the task of investigating sybils be crowdsourced to $10/hour workers on Upwork. This simply moves the challenges FDD is trying to address to other areas: 1. how do you vet the workers on Upwork beforehand? 2. how do you know that the work done by the Upworkers is valid?

I am afraid this approaches severely underestimates the resources and motivations of Red Team; what keeps Red Team from flooding Upwork with applicants that they know will approve their sybils? Or bribing existing workers with $1000 to approve a selected Sybil (given that some bad actors have $100k+ budgets for their grants, targeting a $1k bribe doesn’t seem out of proportion)? The steps that would be needed to take to make crowdsourced work viable are exactly the steps that FDD is currently exploring, with the added incentive that FDD will gain knowledge and retain control of the process.

We always have to think very carefully about what we are optimizing, about the risks and tradeoffs. It is always going to be cheaper in the short run to work with established structures , cut costs, and ignore future growth. Spending time experimenting and exploring uncertain realms, by definition, will always have unknown ROI. But maintaining the growth

@owocki has proposed that GitCoin is an egregore, which means it is a living thing, which means that it needs to be fed and nurtured. I think the egregore that GitCoin needs to become operates differently than a traditional bottom-line business, and this is something that people who are responsible for feeding the egregore (i.e. the stewards) need to consider.

The current FDD proposals are seeking to balance the two objectives of present performance and learning for the future by 1. performing within established knowledge while simultaneously using simulations (as cheap experiments) to gather information about alternative models. The simulations and other models are important, because we are dealing here with a human intelligence network with an intriguing type of information flow. We need models because networks of human intelligence and information flow aren’t amenable to purely logical testing like software programs.

8 Likes

This proposal will be moving to snapshot today and I am requesting stewards to vote for OPTION 1.

While we believe we have the support for option 2 to pass, we still want the stewards who believe we should do a discovery for a Sybil Detection/Passport Protocol DAO to be able to express that interest.

We have done the work, had tough conversations and continue to perform even after losing three people in the budget review process.

As a last request for stewards, I’d like to request that you to consider the following.

  1. All options are more than a 40% deduction from the original request which is more than any steward asked for.

  2. In a grants 2.0 future, we need to consider that FDD is responsible for protecting the legitimacy and credible neutrality of our own Gitcoin community rounds.

  1. As other communities have rounds, they will need to choose their funding stack as well. With that, they will either take recommendations from what we use, roll their own algo, or design new solutions. We can be invested in the generalized FDD solutions as businesses and protocols.
  1. Our work is on a core functions in Gitcoin Grants.

Finally, here is one more way to look at what FDD does based on functions in the funding stack.

4 Likes

Thank you @DisruptionJoe , @ZER8 , @David_Dyor, @Sirlupinwatson for the incredible effort in communicating what FDD is trying to do.

I read your entire proposal the first time, and I did it again. I will vote for Option 2 in the Snapshot.

OPTION 2 - Fund FDD $330,000 and signal they should not perform a discovery 4 of a sybil detection / Passport DAO
(49% reduction from original request)

I believe:

  • innovation is important to any organisation, encouraging it is key
  • prudence is just as important, and I appreciate FDD trying to move towards being a leader in sybil defence

With the changes to Grant 2.0, FDD will need to evolve with it. I think creating some space for that is important while continually executing on FDD’s core functions.

However, I’d like to remind some contributors that Stewards are not your enemy. Therefore I’d like to give some appreciation to @David_Dyor for his carefully written explanation to give context on how we might make better decisions regarding FDD.

Analysis

Comparing both proposals ,here’s the key changes I think happened
Core functions
FDD OS from 124.4k to 95k (-29.4k)

  • removed storytellers (-14.4k)
  • halved data analysis (-15k)

Grants eligibility from 86.5k to 60k (-26.5k)

  • halved ethelo dev (-6.5k)
  • lowered cost for round, reward, GIA which budgeted 60k by 1/3 (-20k)

Data Ops from 50k to 45k (-5k)

  • simply lowered it by 10%

Adding in the Innovation segment
Passport Trustseekers == Sybil Detection DAO from 60k to 60k (-0k)

  • both have same budget @60k

Community Intelligence, probably combination of catalyst and community model from 97.5k to 70k (- 27.5k)

  • guess is catalyst from 37.5k to 30k (-7.5k) because we love schelling points
  • community model from 60k to 40k (-20k), also a 1/3 reduction

My guess is that the “Expansion” is more of a psychological element to nudge stewards to vote for Option 2. which is innovation because its description is so sparse. A bit like fire sales saying original price is $10, 50% discount at $5.

Conclusion

I think FDD has somehow managed to balance most of the original roadmap, cut out some additional items that we’ve given feedback on (thank you for taking in some of mine too!).

Here’s a potential compromise for S15.

I would suggest considering splitting items up and @DisruptionJoe being an advisor to these workstreams
Core to FDD

  • Grants Investigation Workstream
  • Sybil Detection Workstream

DAO innovation

  • Gitcoin Research Workstream - I noticed that research work is being done throughout the DAO including MMM’s research on grants with a case study
  • Passport Workstream - however this should fall under Moonshot/Product team

Yes it will give stewards more to look at. However, consider @kyle 's hypothesis that Thin workstreams might lead to better outcomes. From this perspective, I think splitting innovation from the core function of FDD can help us get our key sybil detection and grants approval process going while we are given more time to discuss and evaluate the direction of innovation we want to go as a DAO, and as FDD.

4 Likes

So for me this one is a very hard decision.

I am leaning towards voting for option 3.

The reasons are that:

  1. I believe FDD, or something like that essential to Gitcoin grants and as such should definitely be something we should have during the grants round.
  2. I am sure that sybil Detection DAO is not something the DAO should spend funds right now under the current market.
  3. I am not confident that the current FDD team is able to deliver on what the DAO needs in the long run. I am not sure if it’s the fault of the team organization, the roadmap they are trying to stick to, or a breakdown of communication between FDD and the rest of the DAO. Something is off.

For Option 2 we are asked to fund “Passport Truthseekers” and “Community Intelligence”. I read the bullet points and explanations of what each of those is. I can’t really judge if it’s useful or not. I don’t have anything to judge by. No results to compare with.

On the other hand I understand that this is what innovation and research is about, but how long can we keep funding research without concrete results?

Then again I see comments by @David_Dyor and @DisruptionJoe here in this thread where they say they are surprised about the misalignment between the DAO and the FDD.

Some of the un-alignment accusations came as a surprise and the FDD should have a season to respond. If you think the scope of FDD drifted please give us the chance to demonstrate how aligned, lean and laser-focused we can be. Help us understand the dao-wide priorities so we can help the dao. Try to guide this crusty teenager before abandoning them. The results will be incredible.

and asking for more time to act on the feedback

It would also leave a 3/4 built skyscraper behind where people could say, “see, GitcoinDAO couldn’t provide a protocol without single points of failure and they wasted all their money on this thing”… but it might just be that finishing building that skyscraper would have removed the single points of failure. (skyscraper = the sybil work done so far)

Which makes me want to help towards aligning more on the goals and giving more feedback. I am really torn, between option 3 and 2, but leaning towards 3.

Even if If I vote for 2, it would still not be a vote for the current way FDD is doing things. We need to reconsider a lot of things. Most important of which are:

  1. Conciseness. FDD has a tendency to go off on a tangent. Every time I look at what you guys are doing, it sounds and looks more and more complicated than the previous time. You also tend to have funny names and very long form descriptions of everything.
  2. Keep it simple. (similar to point 1.). Please keep things simple. Stewards are not your enemies, but if we can’t understand what you are doing, or if we have to spend hours to understand something that should have been straightforward (come on guys it’s not rocket science) we won’t blindly say yes to it.
  3. Alignment on goals. Do one thing and do it well. It seems that FDD is trying to do too many things at the same time. We should have discussion within the DAO about what the FDD’s mandate and goal is again.
  4. Communication. There seems to be a breakdown of communication between you guys and the DAO. This is evident by the surprise you show in the feedback while this was already known between stewards in previous rounds too. Also by the comments in this thread about lack of inclusion in grants 2.0 planning (quoted below).

I would really love very direct feedback from @lthrift (Product Director) @kevin.olsen (VP of eng) @brent (Passport PM) and @nategosselin (Grants PM) to see which specific points are not aligned here. Let’s have a productive conversation!

I am also asking for us to be involved in co-creating a future. I don’t understand why FDD was only involved in the grants 2.0 planning in discovery workshops where the product team for grants 2.0 learned what FDD was currently doing, but not in any future planning where we could have put these ideas forward and course corrected together at an earlier point in this process.

No matter what option is voted for these 4 points need to be addressed.

Since the snapshot vote runs until tomorrow evening and it’s already late here, I will sleep on it and vote tomorrow. But wanted to leave my thoughts here first.

2 Likes

Tl;dr - I will be voting to fund option 2. Grant review/approval (eligibility enforcement, etc.) and the data science work from FDD has been incredibly important for us to continue and encourage the growth of the Grants program.

I want to continue my thanks and appreciation to the FDD team for the ongoing conversation and revisions of this budget. I have appreciated those conversations and the willingness to explore an alternative to the previously proposed focus areas.

As we evaluate the future of Grants and our Identity management tooling (Gitcoin Passport), I am of the opinion that much of the FDD work as previously scoped (before these revisions) have been focused on solving our sybil detection and sybil suppression in the wrong way. This budget proposal and my vote for Option 2 comes with the expectation that we will continue to realign FDD towards partnership in a protocol driven approach (as opposed to an ML based, and human review approach).

A protocol based Sybil prevention approach will make the rounds (for all Grants 2.0 communities) more expensive to attack than they are to defend. Today, our Sybil detection is a very expensive red team vs blue team where we are constantly “whack-a-mole” finding and updating our approach. In the future, defining a cost of forgery for a wallet, which defines eligible matching pool impact will be possible with the protocol. This means, we will let the existing Sybil detection processes fade and shift from how we do things today.

I am interested in funding Option 2 to ensure GR14 still runs (grant eligibility reviewed, sybil detection and “fraud tax” calculated, appeals process on eligibility, etc.). I am also interested in seeing continued data science from Omni as it has been very instructive in helping us shape the future approach. When evaluating past versions of the budget (and option 1), I don’t believe funding other mechanisms should continue and FDD has heard this critique and cut their budget request by ~45% from S13 to accommodate this cut in investment.

One item that I want to flag is that I believe there is an opportunity to better align on how to direct $60k of this budget in closer partnership with the Grants 2.0 and MSC workstreams.

My vote for Option 2 comes with the request that the roadmap for the Truth Seeker’s work is roadmapped and agreed upon with the Grants 2.0 workstream and MSC. They are directing the work for those protocols and I would want to ensure they are aligned with the work to expand our thinking there.

1 Like

How do you vet any new hire? How do you know the work done by any new hire is valid?

The people working for the FDD can be bribed even easier than new people off upwork, as many of them have public twitter accounts, finding out who you hire on upwork would be much harder.

I have found INCREDIBLE people off upwork for Giveth and Commons Stack. In fact, at this point about 1/2 the Commons Stack team was hired off of upwork. None of them would take bribes.

We have to hire off upwork, because we don’t have VC funding, we live month to month on donations, but yet we still scramble to make it happen, and make an impact. In fact, we make a greater impact because we are sending crypto to countries that need it. If you would like help hiring off of upwork, some of the Ops money that was raised can be sent to General Magic, who does our hiring, and they can help find great candidates… and even show them how we do it.

Don’t get me wrong, we need to also have experts like yourself Octopus (you know i love your work) and Z (you KNOW I love his and Blockscience’s work) but where we can, lets stop hiring people from the US and Switzerland, and hire people that can do the same (or often better) work for 1/3 the cost.

The FDD has to cut costs, GTC cannot handle the wild inflation that was voted for this quarter, I can’t believe how much was approved to be spent on Ops and MMM… its ridiculous. IMO more should be spent in FDD than Ops and MMM this quarter… both can have admins from lower income places support their work.

But that ship has sailed, we are where we are and we need to appreciate the market forces and the fact that GTC has no sources to generate demand right now or on the horizon, and tighten up. :-/

But I believe in the FDD team and the expertise they have developed, and am excited to see what they can build this quarter!

I am voting for Option 2.

2 Likes

I am voting for Option 3.

you actually voted for Option 2 though :thinking:

1 Like

So I voted for Option 2 after all. As I wrote here: [Proposal] FDD Season 14 Budget Request - #37 by lefterisjp

I was torn between 2 and 3, but at least from the back and forth with FDD I see a lot of willingness to adopt and adjust and also compared to all other funded workstreams and to their initial proposal the budget ask is much smaller.

Still I ask that FDD considers the points I wrote above for future rounds and for how we can fix the communication problems.

2 Likes

oops! Thx! Fixed it I meant Option 2