The value in testing

A few days ago, Cloudflare vibe-coded vinext, a clone of Next.js.

This has been made possible because of the huge test suite directly "ported directly from the Next.js test suite" (quoting the article). Which raises the question whether the value of a project resides more on the test suite than in the actual production code?

In that case, chances are, the model was already trained on the Next.js codebase. Same goes for the Anthropic vibe-coded C compiler, it was probably already trained on gcc, and compilers are very well documented. In both cases a very solid already existing test suite has been used.

I remembered a podcast I listened to a few months ago, with the founder of SQLite telling the story of the project, which was fascinating. SQLite is open source, but their main test suite, TH3, is proprietary. In the light of this story, I just wondered: are they ahead of everyone? Are they playing 4D chess when everyone else is stuck playing checkers?

It might be an effective strategy to give the trust of open source software without making it copyable. It would be difficult to follow this model for projects that expect open contribution from anyone.

I see similarities between Test Driven Development via AI and a paradigm called "Constraints Programming" (which is from the 1970's). Both focus more on what the solution must satisfy rather than how to solve the problem.

What about just writing specs and let the AI write the tests, then have it implement the production code? I have seen a lot of very complacent or poorly written tests. I would be very vigilant with this approach.

Maybe the near future of SWE is just writing a solid test suite and letting the assistant try multiple implementations until something satisfying emerges? It would solve the "rewriting everything in a new stack every X months" problem (even if I have a different opinion on the topic). In the end, I still think there is value in a carefully crafted and audited production code.