Evaluation of LLMs’ ability to predict intrinsic aqueous solubility based on SMILES representations

Abstract

As far as molecular properties are concerned, solubility is often not the first one that comes to mind. It is a property so fundamental and intuitive that it is sometimes taken for granted. However, it remains a core feature that is continuously being researched, since it affects dissolution, absorption, bioavailability, and formulation. In drug discovery specifically, it is a factor that can determine whether a molecule will be eligible to continue to later stages of a trial. Measuring solubility with lab-based methodologies can be expensive and slow. A technique that avoids the lab would be ideal, in order to screen thousands of molecules at once, and thus avoid inefficient experimental procedures. For that reason, computational solubility prediction has become a field that has attracted researchers to develop effective methods. In this thesis, the aim is to evaluate the ability of various LLMs to predict the solubility of different molecules using SMILES representations as input. The LLMs used are Gemma 3, Qwen 2.5, GPT-OSS and ChatGPT. There are no extra features used to help the LLMs make their predictions, for example descriptors, molecular fingerprints, or any additional training. For this purpose, the AqSolDB dataset is being used, a curated dataset of approximately 10,000 molecules, mostly drug-like. After carefully cleaning, handling duplicates and removing salts and problematic structures, the final dataset of 9,271 molecules was used to create three subsets that were to be used in the experiments. The models were mainly tested on their ability to classify a chemical compound as having high, medium, or low solubility, classes that were created based on specific solubility value ranges. Initially, two pilot experiments were conducted using a 20-molecule dataset. The scope of the two experiments was classification and regression, respectively. The decision to proceed only with the classification task was made, based both on the results and on the practicality of a classification outcome for drug development. Gemma 3 did not proceed to later stages, as the model did not manage to successfully complete the task. In a 50-molecule experiment, Qwen 2.5 showed heavily biased behavior leading to the discontinuation of that model as well. On the contrary, ChatGPT and GPT-OSS produced reasonable results, with ChatGPT achieving as high as 0.80 accuracy in the 5-shot case. Generally, it was observed that the 5-shot prompt type produced the strongest results for both models. A majority vote experiment on the same 50-molecule dataset followed, which aimed to reduce the effect of unstable individual predictions and thus increase the reliability of the models. However, it did not seem to improve prediction accuracy. Since 50 molecules were thought to be a relatively small sample size, a larger subset containing 300 molecules was utilized to check if the obtained results were stable. The trends noticed so far also appeared in these runs, with ChatGPT continuing to lead in performance. The final experiment included a second agent that assigned a confidence class to the first agent’s prediction, essentially evaluating the quality of that prediction. The results showed that high-confidence predictions were significantly more reliable, but confidence was not always optimally calibrated. Overall, the results suggested that LLMs can act as a practical solubility prediction tool taking SMILES strings as input, but with limitations. A level of bias does exist, which must be considered when using them in real world applications. Furthermore, classification appears more reliable than regression for this setup, but even in this case the output may be better treated as a supportive signal rather than a sole decision-making factor.