Is it possible for a self-extracting file to run malware on Linux?

welcome to this SE site.

First of all, you may have a file with the most malicious content, stored on any OS, making no harm. The activation of the malicious behavior (from now on, ‘infection’) would require it to be executed, either directly or indirectly.

The eXecute permission unix systems (including Linux) mark whether a file can be run or not. As you would need to set it as executable, it may prevent some cases where you could inadvertently run a file you didn’t intend to. Note however it doesn’t completely prevent them from being run. This is specially the case for interpreted languages, where not having the execute bit would for instance not let you run ./ but you could run them as bash, and similarly with python, perl, etc. (you could as well run ELF binaries that way using ldd, but not on recent versions)

(NTFS also has an execute permission, although it is rarely used)

A direct execution would involve the user running the program (maybe thinking it was doing something else, like opening a pdf invoice). An indirect execution would involve code being run by another program. For example, a vulnerability on the pdf viewer could make it run malicious code when the user opens a specially-crafted malicious pdf. There the execution permission of the file itself won’t matter, since it’s the actual pdf viewer the one which ran the code.

Moreover, this could even lead to execution of files that the user didn’t interact with, such as the exploit happening automatically when opening the folder, as it was parsed by a vulnerable reader to produce a preview/thumbnail. Or an entivirus that would ran a virus when trying to scan it.

Is that enough to prevent any malware from being installed by some self-extracting file?

No. The best prevention would be not to run such executable at all. Not even if it’s “just” a self-extracting file.

Is there anything that can be done to increase the security on a Linux machine?

You could enable SeLinux / AppArmor / run specially untrusted processes in a sandbox / virtual machine. And all the usual recommendations: Do not run untrusted programs or random instructions you found on the web, separate yourself form root account (obviously), keep the system and programs updated (Linux distributions makes this very easy), have working backups that would let you recover if anything bad might happen. Plus, acting cautiously and sensible goes a long way.

Additionally, you could further increase its security by unplugging it, locking in a safe, and storing buried 20 feet under the ground in a secret location (Dennis Huges), albeit you are probably unwilling to go that far.

Recently I came across this extension which can save a webpage as a self-extracting HTML file.

When talking about self-extracting files, they are usually about executable files. They don’t need that the receiver has a program able to unpack them, since when run as a program, they are able to unextract themselves. The counterside, is that it requires running that program, as opposed to a local unpacker which would (hopefully) have been installed from a trusted source. (Note: it is usually possible to open self-extracting compressed files with the appropriate program, without having to run them)

In the case of that extension, opening those pages is safe. It works by creating a mixture of html page and zip file, that when opened, loads the contents with javascript. Since it runs inside tha browser, it’s not more unsafe than opening a web page stored locally. (Note: installing the extension would still require the usual analysis of determining if that’s one that could be harmful)

And aren’t formats like EPUB and CBR just archive formats with a specific structure?

Yes, they are archive formats.

Can malware be hidden in these file formats as well?

Malware files could be hidden there. Or a love letter. Or instructions for a spy ring. The point is if that would be dangerous in some way. Most probably not. They would need to exploit a vulnerability in the EPUB/CBR reader, which is the one that would process that file. Otherwise malware stored there would be harmless.