The benchmark attracts on 117 revealed papers and patents and consists of 761 distinct molecular constructions throughout 99 issues.
On the xhigh reasoning setting, GPT-6 Astra earned 95.4 p.c of the out there credit score for reconstructing molecular constructions. Nonetheless, the mannequin earned solely about half the out there credit score for choosing compounds and predicting their numerical properties.
The findings spotlight a niche between figuring out chemical constructions accurately and understanding how adjustments to these constructions affect a possible drug’s conduct.
Medicinal chemists routinely steadiness competing properties when creating drug candidates. A compound could successfully block a disease-related protein however carry out poorly when absorbed by the physique or break down too shortly.
Enhancing one attribute also can weaken one other, making it troublesome to determine compounds that meet a number of necessities concurrently.
MedChemBench evaluates 4 capabilities: molecular construction reconstruction, scientific reasoning, compound rating and choice, and numerical property prediction.
Certainly one of its issues attracts on analysis into inhibitors of PRMT5, a protein studied in most cancers drug discovery.
Fashions obtain molecular constructions and experimental outcomes from earlier phases of the analysis. They have to predict how three associated compounds will carry out when it comes to PRMT5 inhibition in cells, permeability throughout a cell barrier, and breakdown by liver enzymes.
Later experimental outcomes, withheld from the fashions, revealed competing benefits. Compound 23 inhibited PRMT5 at a decrease focus, whereas compound 24 crossed the cell barrier extra readily and degraded extra slowly within the laboratory assay.
The duty exams whether or not fashions can acknowledge that stronger exercise in opposition to a organic goal doesn’t essentially translate into higher absorption or metabolic stability.
Molecular accuracy stays essential
The benchmark additionally examines whether or not fashions can reconstruct molecules precisely from chemical drawings, together with their atom connections, ring constructions and stereochemistry.
In a single instance, fashions have been requested to assemble a compound whereas preserving an 18-atom ring.
GPT-6 Astra reconstructed the construction accurately in all 10 responses on the highest reasoning setting shared by the evaluated fashions. In distinction, GPT-5.6 Sol produced a smaller, 15-atom ring, whereas GPT-5.6 Luna generated disconnected molecular fragments.
Such errors can change the id of the compound being analyzed, undermining subsequent predictions about its properties.
Regardless of its robust structural efficiency, the mannequin struggled extra with compound choice and numerical predictions. These duties require connecting molecular adjustments with a number of experimental measurements somewhat than merely reproducing a chemical construction.
The researchers evaluated 11 fashions throughout totally different reasoning settings. Larger reasoning effort usually improved efficiency, though substantial variations remained between fashions.
The group cautioned that the outcomes don’t set up whether or not fashions can uncover new medication or enhance laboratory choices. The benchmark depends on historic analysis, and prior publicity to revealed sources can’t be dominated out. Fashions additionally accomplished the duties with out instruments, suggestions, or new experiments.
MedChemBench due to this fact presents a solution to determine progress and remaining weaknesses in computational reasoning for medicinal chemistry, somewhat than demonstrating that these techniques are prepared to exchange drug discovery analysis.

