← All insights

Insight / Research and infrastructure

Published

A place for open research, honest benchmarks and shared compute

A proposal for a community space where people can share papers, reproducible AI setups and measured infrastructure needs before exploring pooled compute.

Researcher working with equipment in a laboratory
Photo: Nathan Rimoux / Unsplash

There are many places to publish a paper and many places to post a benchmark screenshot. It is harder to find a record that connects a research question, the exact model and machine used, the measured result, the limitations, and the compute needed to repeat or extend the work. A useful community space could make those connections visible without claiming ownership of the research it hosts.

This is a proposal, not an announcement that Paradox Design already operates a paper repository or shared GPU service. The first version would be a place to describe work and link to durable artifacts. Collective compute would come later, only if there is demonstrated demand, a workable operating model and clear governance.

Start with records that a reader can reproduce

A paper page should identify the authors, title, version, date, subject, abstract, licence and a persistent link to the primary artifact. It should distinguish a preprint from a peer-reviewed publication and show corrections or withdrawals. Rather than replacing established repositories, the space could link to OSF Preprints, Zenodo or an appropriate disciplinary archive. A link to a DOI and a checksum for important files is more durable than a screenshot of a result.

A benchmark page needs more than a token-per-second headline. Ask for model name and exact revision, quantisation, runtime version, prompt length, generated length, context, concurrency, hardware, driver, power limit and measurement method. Include time to first token, sustained output, memory peak, quality checks and failures. If a result uses private data, publish a redacted methodology and synthetic reproduction case instead of the data itself. Our local AI performance checklist gives a starting measurement template.

Let people correct their records. A new driver, model quant or benchmark script should create a new version, not silently overwrite the old comparison. Moderation should distinguish a failed reproduction from misconduct, and give contributors a way to respond.

A contributor could also describe the operating structure that made the work possible: an individual workstation, a university research group, a company, an association or a co-operative. That field should explain who owned the hardware, who paid for operation, how access was approved and which agreement governed data or results. It should not require contributors to publish private contracts or imply that one legal structure is right for every researcher. These narratives could be as valuable as the benchmark numbers for someone planning their first serious setup.

Turn demand into an infrastructure question

If a researcher says they need “a big GPU”, the useful next questions are more specific. Which model and precision? How much memory is required for weights and context? Is the work training, fine-tuning, batch inference or interactive use? How many users will run at once? What data may leave the organisation? What deadline and uptime matter? Can the work pause, checkpoint and resume?

Those answers can produce an indicative compute profile and a budget range. They can also reveal that the work would be cheaper or safer on an existing university cluster, a local workstation, a managed cloud service or a smaller specialist model. A community platform should not steer every request toward a pool it happens to sell. Our local infrastructure calculator can frame the demand discussion, while the Strata article illustrates why supported model and memory behaviour matter as much as headline parameter count.

Shared compute is a governance project

Pooling hardware sounds simple until several people need the same accelerator, private datasets are involved, or a machine fails during a deadline. Before taking funds or workloads, define membership, scheduling priority, fair-use limits, identity and access controls, data separation, logging, backups, incident response, maintenance windows, electricity and insurance. Explain what happens when a member leaves and how their data and credits are returned or deleted.

There are several possible structures. business.gov.au’s structure guide explains that a company, partnership, co-operative and other structures differ in control, cost and liability. A co-operative may suit genuinely member-owned capacity, but it has membership and governance requirements. A company may suit an operated service with clear commercial responsibility. An informal group may be enough to compare benchmarks but a poor vehicle for holding shared funds or sensitive datasets. The right structure would require accounting, legal and community advice for the actual activity and jurisdiction.

The project also needs content rules: authors keep ownership of their papers, contributors choose a stated licence for uploaded material, sensitive or infringing content is handled promptly, and benchmark sponsors disclose relationships. Do not put an unpublished paper, private model or customer dataset into a public pool because the interface makes uploading easy. The concerns in our article on student work and Turnitin apply here too: participation should be informed and reversible where the platform can make it so.

Build a narrow first version

A useful first release could accept a paper link and a structured benchmark record, allow a contributor to claim and update their own record, and let readers compare setups using explicit filters. It should not promise peer review, paper permanence or compute allocation before those capabilities exist. Measure whether people submit reproducible records and whether others can learn from them.

Only then test shared capacity with a small, permissioned pilot: known participants, a limited workload, clear cost accounting and an exit plan. Publish the rules and the limits alongside the results. If there is no sustained demand or the governance burden is too high, the research and benchmark directory can still be valuable on its own.

An open research space succeeds when a reader can understand what was tried, what worked, what failed and what it would take to repeat it. The future compute layer should be earned by that trust, not assumed in advance.

External project website

Open this project site?

is hosted at . It will open in a new tab.