Hi everyone, thank you for your generous patience in waiting for revised matching calculations. We’re happy to finally have the updated results, which you can find here. We’re also extremely grateful to everyone who commented, provided feedback, criticism, questions, and more. It shows the community truly cares and wants to help make Gitcoin great.
I also want to specifically thank @epowell101 @PouPou @umarkhaneth and many others from Gitcoin and ODC who contributed their time and skills to this analysis.
In aggregate over 20 different tests were run on all transactions from the 5 rounds, many of which were tools built in the ODC hackathon. These tests considered factors like the seed funding of a wallet, number of transactions, interactions with other donors, interactions with various dapps (DEX’s, Tornado, Disperse, etc), likely airdrop farmers, clustered/consecutive transactions between different wallets, and more. Addresses were also cross-referenced with externally maintained Sybil/bot lists, as well as checked for interactions with addresses on those lists.
We then backtested the output of all these metrics against previously identified Sybils, suspect addresses discovered manually in this round, and checked the correlation with passport scores. Many of these tests did not appear to be significantly correlated with suspected Sybils, or they produced a large number of seemingly false positives. As a result, we decided on a select few tests to use for the final Sybil squelching, which appeared to be most effective at catching likely Sybil attackers while minimizing the number of honest users caught in the crosshairs. These tests included:
Suspicious seed - verify that the first incoming transaction (regular transaction and not internal transaction) is seeding the wallet = providing gas to the wallet. If the first transaction is not a seed, it is suspicious. This happens when the first normal transaction is an outgoing transaction. It is possible if it was funded from an internal transaction.
Clustered addresses - The LCS test runs on any address that has less than 10 transactions. It then compares the transaction history of that address with all other addresses looking for similar consecutive transactions. If the number of consecutive transactions is above 3 it raises the flag.
Less than 5 transactions - any fresh wallet with under 5 total transactions flagged.
Detected by stakeridoo (ODC) - Detected sybils using the fly trap method. He analyzed addresses that have received an airdrop and he managed to find clusters of addresses that sent the funds back to the main wallet demonstrating the connection between these wallets. More details here: CharmVerse - the all-in-one web3 space
Detected by doge (ODC) Analysis is here: https://github.com/DistributedDoge/sybilscope/blob/master/sus_on_chain.ipynb
He used flagged wallet by hop protocol to detect Sybils and as well as jacard similarity using these flags for similar donation patterns
These results were combined with the initial Sybil tests described in the first post of this thread, along with some additional manual reviews and flagging.
When looking at the results sheet, please note that the “Total Received USD” amount is what is reported by the platform before this additional analysis and revisions. However that USD amount does by default remove donations less than $1 and donations from failed passport scores.
Sybil/fraud detection and prevention on a decentralized protocol has certainly proved to be more difficult than on our previous (centralized) grant platform, where we had more and different data to leverage. That said as we build more robust on-chain analytical tools, this process will become easier and faster. I am (personally) excited to explore features like a “trust bonus” system (ie. a variable trust coefficient rather than a binary yes/no). This would mean the highest “verified” users have their donations matched the most, so building up a strong reputation over time is important. Future experimentation with pairwise bonding, plural QF, and other anti-collusion mechanisms may also prove to be effective at limiting bad actors.
Gitcoin will likely share a more detailed analysis of the data from the Beta Round and the calculated results soon, so keep an eye out for that. I believe @ evan will also be sharing more details and takeaways from ODC’d perspective. We look forward to collaborating with the ODC and others in the community more going forward.
From this point we still plan to wait 5 days for review and feedback on the revised results before moving to a Snapshot vote, which if passed will be followed by payouts shortly after. If you have any questions, please don’t hesitate to reply in this thread. Thanks all!