The resolution of microscopy images is crucial for their analysis. Conventional microscopy techniques, such as wide-field, two-photon, and confocal microscopy, are all resolution-limited as their resolution is determined by the physical properties of light at approximately 250 nm. The limitation is problematic because it exceeds the size of most multi-protein complexes, which typically ranges from 25 to 50 nm in diameter, rendering them indistinguishable when visualized. This is particularly important for projects that evaluate colocalization and require precise identification of subcellular structures marked with different colors that are in close proximity to one another.
In the mid-2000s, scientists successfully overcame the diffraction limit by introducing advanced microscopy techniques that enhance image resolution to approximately 100 nm. This breakthrough made it feasible to study the subcellular localization of proteins. The term super-resolution microscopy was coined to describe these techniques that allow visualization of structures below the diffraction limit of light. More recent advancements in microscopy instrumentation have enabled even higher resolutions in the range of 25 to 40 nm.
Despite their higher resolution, super-resolution microscopes have several drawbacks:
Recent advancements in Machine Learning (ML) and Deep Learning (DL) have introduced an alternative to achieve super-resolution in microscopy images. By utilizing ML models, it is possible to enhance images from conventional microscopes, yielding high-resolution results comparable to those obtained from super-resolution microscopes.
The Super Resolution model for CoLocalizer was developed with three key requirements:
1. Efficient architecture.
The model should have an architecture that is well-suited to our problem. For implementing resolution enhancements to fluorescence images with colocalization, we focused on unsupervised learning using a Generative Adversarial Network (GAN). GAN is a ML framework based on minimax game theory that simultaneously trains a generative model and an adversarial model. This network is designed to generate specific distributions and performs well with significantly less training data, even without paired training sets.
2. Accessible data sets.
High-quality training and testing image data sets are crucial for creating an effective ML model. However, obtaining and managing such sets can be challenging and prone to error. We aimed to use minimal image data without the need for extensive time and effort on annotation.
3. Real-time performance.
The model should provide near real-time performance to deliver visual feedback on enhancements made to the images.
We decided to create an ML Super Resolution model using unsupervised learning with a GAN network. To ensure accurate image-to-image transformation, we employed a content-preserving cycleGAN. This approach utilizes cycle-consistency learning to convert images from conventional to super-resolution without requiring paired data. Consequently, we achieved performance levels comparable to supervised learning while using fewer training images.
The model consists of three independent cycleGAN-based networks for specific image restoration tasks:
In simpler terms, the model can be represented as follows:
In our model training process, we used 1800, 73200, and 3900 sets for the Denoising, Axial Resolution Restoration, and Super-Resolution Reconstruction networks, respectively. All sets comprised unpaired low and high-resolution images.
After applying the ML Super Resolution model, the resolution of the images increased threefold. Tests revealed that the Peak Signal-to-Noise Ratio (PSNR) of the restored images averaged an increase of 7.5 dB compared to the original images. Quantification of super-resolution artifacts indicated a resolution-scaled Pearson correlation (RSP) of less than 1 and a low-resolution scaled error (RSE) of less than 10 (on a scale of 255), suggesting a very low degree of distortion.
The ML Super Resolution model in CoLocalizer Pro significantly enhances image resolution while preserving and improving details. This advancement will transform colocalization analysis by revealing structural details that were previously obscured, making the process more convenient and affordable. It will also be valuable for re-examining images from past experiments that cannot be replicated.
While the model was specifically designed for fluorescence images with colocalization, it can effectively super-resolve any fluorescence microscopy image with similar results.
We plan to further develop our model and its implementation to ensure the most reliable outcomes and the best possible user experience.
If you have any questions, please reach out to us through our Support Page.
