I found a fix to this.
This graphics card has semi-passive mode on by default. It only reacts on how hot your temps are. Even in the MSI Afterburner, the fan speed is locked and could not be overridden. Which is why, I tried searching online for possible fixes for this and found nothing but one thing.
Aorus Engine.
https://www.gigabyte.com/Support/Utility/Graphics-CardAfter installing this, I figured out that the fans are set to Auto and Semi-Passive mode.
Then I just simply set the fans to Manual and Always ON. This problem never happened again.