apache 2.2 – Restrict Direct IP Access To Website Without Affecting Other Sites

I have several sites hosted on my server. If you directly type in the IP address to the server it will take you to one of the sites, I wanted to restrict this behaviour so I used the following fix:

<VirtualHost 192.168.1.1:80>
ServerName 192.168.1.1
Redirect 403 /
ErrorDocument 403 "Forbidden"
DocumentRoot /var/www/html
ServerAdmin support@siteadmin.com
UseCanonicalName Off
#UserDir disabled
</VirtualHost>

And it worked for restricting direct IP access, however now my other sites are now unreachable and display the forbidden message.

How can I change this so that direct IP access is still forbidden, but my other sites are still able to be reached using their Domain Names?

Here is my full config file:

 <VirtualHost *:80>
    ServerName admin.example.com
    Redirect permanent / https://admin.example.com/
</VirtualHost>

<VirtualHost 192.168.1.1:80>
   ServerName 192.168.1.1
   Redirect 403 /
   ErrorDocument 403 "Forbidden"
   DocumentRoot /var/www/html
   ServerAdmin support@example.com
   UseCanonicalName Off
   #UserDir disabled
</VirtualHost>

<IfModule mod_ssl.c>
<VirtualHost *:443>
        ServerName admin.example.com
        ServerAdmin admin@example.com
        DocumentRoot /var/www/html

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        Alias /static /home/user/example/static
        <Directory /home/user/example/static>
                Require all granted
        </Directory>

        <Directory /home/user/example/example>
                <Files wsgi.py>
                        Require all granted
                </Files>
        </Directory>

        WSGIScriptAlias / /home/user/example/example/wsgi.py
        WSGIDaemonProcess example python-path=/home/user/example python-home=/home/user/example/venv
        WSGIProcessGroup example

SSLCertificateFile /etc/letsencrypt/live/admin.example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/admin.example.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>