Is it possible to extract encryption keys from software executing on your own system?

You don’t say that you’re dealing with malware, but I will assume you are.

If the application is doing encryption, then of course it has the key. The fundamental approaches are to either analyze the application binary on disk, or to let it run and analyze the running binary. But be aware that malware authors do not want their applications to be reverse-engineered, and they make that very difficult to do; so you are venturing into the field of “malware reverse engineering” which apparently has a median salary of $125,000 USD, which tells you a bit about how specialized the skillset is.

So when you reverse engineer the malware you may find for example that they use binary obfuscation techniques to make the binary very difficult to understand, for example to figure out what is code and what is data (ie the key you’re looking for). Another technique in this direction is whitebox crypto which aims to, for example take a traditional AES implementation and key, and combine them into a custom algorithm, so there actually is no key, just some very complicated code. You may also find that your malware sample is designed to detect when it is being monitored display completely different behaviour when it is being monitored (let’s call this the Volkswagen maneuver).

Finally, as @mti2935 points out,with some early ransomware it was actually possible to reverse all this and extract the encryption / decryption key from the malware sample. Then the malware people got wise and started using asymetric crypto where you need the public key to encrypt and the private key to decrypt. The public key would be embedded in the malware, and the private key would be kept on the command&control server (c&c). Therefore the malware contains what it needs to encrypt, but not matter how hard you look, the keys to decrypt are not in there.

To be even more clever, when the malware starts up, it can contact the c&c to get a unique public key so that they can demand separate payment for each instance of the malware because they are all using different keys.

In summary: Not easily. Good luck. If this you are asking about ransomware, sounds like you need to hire a malware specialty firm.