Model Submissions GG24 Deep Funding

[omnianalytics]

Omniacs.DAO — Using AI-Guided Search in Deep Funding Level III

Background Context and Motivation

At this point in time The Omniacs squad has been grinding on Deep Funding related topics for over a year. If you don’t believe us, check out all our old submissions here, here, here, here, here and here. By now you know we like to “try stuff” and this “Season” of Deep Funding was no different. In the past, we’ve followed the rules, bent the rules a tad, and this time we decided our new angle would be get a subscription to ChatGPT and Grok and let them loose on this problem. After discussing the structure of the contest with ChatGPT early on, both it and Grok became convinced that a reasonable AI-native approach was to treat the leaderboard as a sparse feedback signal and run a disciplined search process around a strong public baseline. Translation, it wanted to leaderboard hack a bit, and we didn’t stop it. That became the motivation for what it described as “gradient descent with guard rails”. We didn’t want to get in the AI’s way, so we just let it cook, even if it wasn’t exactly taking the standard approach. Did it work? For Level III not really, but for Level I and Level II, at the time of writing we were first and third, respectfully (this is all ignoring the effect the final hold out data will have, but for now we’ll enjoy the bragging rights). Over the course of our write ups for Level I, Level II and Level III, we’ll describe the results of letting AI loose on the problem.

Admittingly, Level III is going to be kinda straight forward and bland because the AI really couldn’t catch a good vector and we didn’t have as much fun as we did for Level I and Level II. We’ll have a more entertaining talk about those levels in the coming weeks, but for right now we’ll just have the AI walk everyone through its approach for this. Later, we’ll also try to talk a little bit about our experience doing sybil detection on the leaderboard and interacting with Seer’s prediction markets.

Level III AI Cookbook

We started from the best public structural prior we could find, made controlled perturbations, observed how the score changed, and used that as directional information for the next step. Rather than trying to build one grand model all at once, we asked what an adaptive model would do if it had to learn from limited external feedback and update its beliefs incrementally.

This process eventually got us to a score of 0.3428.

Phase 1: Establishing a Strong Baseline

We first compared the official sample-style submissions against the stronger public baseline derived from the published dependency seed weights. That quickly showed that the public seed-based baseline carried much more signal than the generic sample file and gave us a much better starting point.

Phase 2: Testing Broad AI-Informed Reweightings

Our first instinct was to use broader AI-style reasoning to reinterpret the whole dependency matrix at once. Those early attempts generally underperformed, which suggested that the hidden objective was rewarding structural priors already embedded in the public baseline more than our first-pass global heuristics.

Phase 3: Switching to Gradient Descent with Guard Rails

At that point, we reframed the task as an iterative search problem. Each submission became a controlled perturbation of the current best file, and each leaderboard result became a directional signal telling us whether a particular move in weight space was helping, hurting, or doing nothing meaningful.

Phase 4: Finding the First Reliable Direction

The first useful progress came when we identified a narrow family of edges that seemed slightly over-credited in the baseline. Small penalties on that family improved the score, while moving in the opposite direction hurt it, which gave us the first real locally useful gradient signal.

Phase 5: Increasing Step Size

After a while, the small moves stopped producing meaningful score variation. We concluded that the search steps were too small to resolve clearly against the leaderboard, so we began taking larger but still structured steps, which produced a much clearer series of improvements.

Phase 6: Localizing the Search to a Small Winning Core

A later overshoot helped reveal that only a small subset of repos was carrying most of the gains. From there, we narrowed the search to a focused set of responsive repos, ran selective line searches and controlled overshoots on that subset, and that path eventually brought us down to 0.3428.

What We Think Worked

A few things seem especially important in hindsight:

• starting from the strongest public structural prior rather than the generic sample submission,

• treating the leaderboard as a limited but useful feedback mechanism,

• making structured perturbations instead of arbitrary changes,

• increasing step size once a promising direction was found,

• and narrowing the search once it became clear that only a small subset of repos was driving most of the improvement.

Omniacs.DAO — Using AI-Guided Search in Deep Funding Level I

Executive Summary

We entered this round with grok_45 as champion (loss = 0.3626). Through deliberate sparsity + block-level coordinate ascent we drove the loss down to 0.3263 — a 0.0363 improvement (≈10% relative gain) in the final stretch of the contest.

The breakthrough came from discovering that zeroing the entire long-tail (Block 9 and everything after dappnode/DAppNode) consistently outperformed full vectors. From that sparse baseline we applied clean relative boosts only to Block 4_Languages_Security and renormalized the non-zero weights to sum = 1.000000. The result is a clean, fully reproducible sparse champion that significantly beats every prior full-vector model we tested.

Approach

Phase 1: Sparsity Discovery (the game-changer)
Early accidental truncation (missing tail weights treated as 0) produced surprisingly strong scores. We formalized this into a deliberate “longer_sawed_off” pattern: exact grok_45 weights for the first ~69 repos, then blank (zero) weights for every repo starting at intellij-solidity/intellij-solidity through the final entry. This single change alone moved us from 0.3626 → 0.3275 and became our new baseline for all further optimization.

Phase 2: Block Coordinate Descent (focused on the hottest lever)
We grouped the 98 repos into the 9 architectural blocks previously identified, but quickly zeroed in on Block 4_Languages_Security (the 8 language & security libraries) as the dominant positive gradient. All subsequent candidates were generated by applying a relative boost only to those 8 repos on the sparse baseline, then renormalizing the non-zero portion of the vector to sum = 1.000000 (zeros left blank to match our winning submission format).

Phase 3: Delta Mode + Controlled Probing
Once sparsity was locked, we switched to strict delta mode:

• Small relative perturbations (±2% to ±4% steps around the emerging sweet spot)

• Whole-block only (never per-repo)

• Full renormalization after every change

• Kept the exact same zero-tail pattern on every file

This allowed dozens of clean iterations while staying well inside context limits. We also tested a brief Block 1 + Block 4 combo; it regressed sharply, confirming we had already found the global sweet spot for this contest.Key Results

File	Loss	Notes
grok_45 (full)	0.3626	Starting champion
grok_45_longer_Sawwed_off	0.3275	Sparsity breakthrough
grok_69 (+18% Block 4 sparse)	0.3264	First sub-0.3270
grok_72 (+20% Block 4 sparse)	0.3263	Final champion
grok_71 / grok_73	0.3264–0.3265	Tight plateau around sweet spot

Key Insights / What Worked

• Sparsity is king: Zeroing the long-tail removed noise and concentrated the entire weight budget on high-signal repos. The jury clearly penalizes diffuse probability mass on low-impact projects.

• Block 4_Languages_Security was the single strongest lever across the entire contest. Moderate boosts (≈+18% to +22%) in sparse mode produced the tightest cluster of record scores.

• Block-level delta perturbations + the leaderboard as a real-time gradient oracle proved far more efficient than per-repo fiddling or large random jumps.

• The “longer sawed-off” format (exact zero pattern) was perfectly reproducible and consistently beat full vectors by 0.03–0.04 loss.

Huge thanks to the Grok team for the real-time renormalization engine, perfect delta-mode math, and instant CSV generation that let us iterate at contest speed.We are extremely satisfied with 0.3263 and believe this sparse Block-4 champion is highly competitive for the final Deep Funding Ethereum round.

Omniacs.DAO — Using AI-Guided Search in Deep Funding Level II

I think we’ll just freestyle what we did for this one instead of a long drawn out explanation. For the originality round we utilized a “diffusion approach” where we submitted random weights from a Dirichlet distribution then tracked how those individual changes in the weights affected the score. We then tried all “obvious” weightings such as: “all 0s’”, “all 1s”, “all .5s”, alternating 1 and 0s, and in blocks. This quickly exposed the back end scoring formula, which allowed us to get a top score with a submission of all .76s.

With that lead, we continued on with our diffusion approach, which yielded this pretty graphic.

The figure above shows the repo weights as columns going from highest (worst) to lowest (best) scores. You can see how the repo weights converge ultimately to the weights that were good enough to get us the top score…

…that’s until the weights were released and 0’s out the board.

Here is some behind the scenes graphics of the progression of our submissions.

For the more technical details of how we used a regression analysis to determine the weights, you can view the Chat GPT write up below.

Our submission to the originality scoring challenge ended up being much less of a standard modeling exercise than we expected at the start.

We came in assuming this would mostly be a straightforward supervised learning problem: fit a model on the historical submissions, estimate how each repo weight influences score, optimize the fitted surface, and submit the resulting weights. That worked at the beginning, but only up to a point. As the competition progressed, we learned that the best path was not simply “fit a better regression.” Instead, the contest gradually pushed us toward an iterative leaderboard-guided search process where the real challenge was understanding which kinds of moves the scorer would actually reward.

Executive Summary

• We began with regression-based approaches designed to estimate how repo weights affected the score.

• Early on, rank deficiency and instability made plain OLS unreliable, so we moved to ridge and additive quadratic ridge models.

• Local weighted quadratic models produced a major breakthrough and got us from the mid-range of the leaderboard down into the low score region.

• Once we approached the best basin, many model-driven directions stopped helping. At that stage, broad optimization became less useful than staying close to the best observed submissions.

• Our final improvement came from a very simple idea: interpolate between the best elite submissions rather than following a newly estimated gradient.

• That final interpolation-based search produced our best result.

Phase 1 – Build the regression-ready dataset

The first important step was getting all prior submissions into a usable format. Each study became one row, the score became the target, and each repo weight became a predictor column. This let us finally look at the problem as a structured response surface rather than a pile of isolated CSVs.

Once we had that, the initial question was straightforward: can we learn the score as a function of repo weights?

Phase 2 – Linear models and the rank problem

Our first pass used linear regression. This gave us a baseline, but it quickly became obvious that the design matrix was underdetermined early in the contest. Coefficients were unstable, sign flips were common, and the raw OLS optimizer tended to push weights to corners in a way that did not match what the scorer rewarded.

Ridge helped stabilize the linear fit, but it did not solve the deeper issue: the scorer was not behaving like a simple linear function of the repo weights.

That pushed us toward nonlinear structure.

Phase 3 – Additive quadratic models

The next major improvement came from additive quadratic models of the form:

[
\hat y = \alpha + \sum_j \beta_j x_j + \sum_j \gamma_j x_j^2
]

This turned out to be a much better approximation than the linear model. In particular, it captured an important empirical fact we kept seeing in submissions: many repos were not best at the extremes, and the scorer seemed to penalize some values that were too low or too high.

Quadratic ridge gave us our first really useful optimizer. It did not perfectly describe the scorer, but it was good enough to generate directions that materially improved our score.

Phase 4 – Local weighted quadratic ridge

The biggest breakthrough in the contest came when we stopped treating all prior studies equally and instead fit local weighted quadratic models centered on the current best submission.

This changed the problem from “what is the best global weight vector?” to “what does the scorer seem to want near our current winner?”

That local perspective mattered a lot. It produced the direction that moved us from a good submission into a much better one, and then improved it again. This phase was where the contest stopped feeling like generic model fitting and started feeling like a controlled optimization loop:

1. center on the current best file

2. fit a local weighted quadratic model

3. generate a small family of candidate steps

4. submit them

5. keep the best and repeat

That process worked extremely well for a while.

Phase 5 – When more modeling stopped helping

Once we got close to the best region, something interesting happened: many sensible model-based directions stopped working.

We tried:

• broader local quadratic refits,

• sparse block search,

• boundary micro-adjustments,

• good-submission manifold search using PCA,

• direct repo-by-repo optimum submissions.

Most of those got worse, sometimes much worse.

The lesson for us was that by the time we reached the low-score regime, the problem was no longer “find a downhill direction.” The problem had become “stay inside a very narrow good basin.” Smooth moves away from the best file often made score worse, even when those moves looked justified by a fitted model.

Phase 6 – Elite interpolation

The final improvement came from abandoning the idea that the next best file had to come from a newly estimated optimum.

Instead, we asked a much simpler question: what if the best solution lies between the best submissions we already found?

That led us to an elite interpolation strategy. Rather than follow a new regression direction, we blended the top files directly. This turned out to be the most robust late-stage method we tried.

The top-2 elite blend outperformed the broader elite centroid, which suggested that the best region was not “the center of all good files,” but more likely a very narrow line segment between the best two.

That was the method that ultimately produced our final best score.

What we think the contest taught us

A few takeaways stand out.

First, identifiability matters, but only up to a point. Early on, improving rank and stabilizing the regressions was necessary. Later, however, the limiting factor was no longer identifiability. By the end, the additive quadratic model was well identified, but that did not mean it was the right optimizer for the true scorer.

Second, local modeling was much more useful than global modeling. The best improvements came from asking what worked near the current winner, not from optimizing the whole surface at once.

Third, the scorer appears to reward a delicate coordinated balance across many repos. That is why single-repo logic and sparse block moves mostly failed near the optimum, while tiny interpolation moves between already-good submissions continued to work.

Final Thoughts

Our final process ended up looking less like standard predictive modeling and more like an empirical search procedure guided by statistical models, leaderboard feedback, and a willingness to pivot once an approach stopped producing gains.

The progression was roughly:

• build the regression-ready matrix

• diagnose instability and rank issues

• move from linear to quadratic ridge

• localize the fit around the current winner

• use local models to find productive directions

• stop trusting broad model moves near the optimum

• finish with elite interpolation inside the best observed region

In other words, the final score did not come from one elegant model. It came from treating the contest as an iterative optimization problem, learning what kind of moves the scorer actually rewarded, and adjusting our strategy as the search landscape changed.

Appendix - See Prediction Markets

There wasn’t much to add about the Seer experience that we didn’t touch on last time. One clear piece of advice would be:

1. Provide additional visibility into the automatic trading algorithm so that when you are about to trade, you get an estimate of the change in balances of the individual repos. I know this is hard because there are so many, but it’ll help save traders who come to add to their positions only to have the automatic trading algo sell tokens they didn’t want to sell or buy tokens they didn’t intend.

2. Related to above, there should have been an easy way to buy more of the tokens you held, despite the probability. It was confusing, but in order to manipulate what you could buy or sell you had to manually manipulate the weight file, which is counter intuitive.

OvVerall the user interface was fine and there weren’t any obviously glaring bugs.

Keep up the good work Seer!