"Allow USB debugging" pops whenever "Always allow in this computer" is checked in the automation test

We run automation tests on mobile devices using Appium, but testing fails because the "Allow USB debugging" pop-up is displayed even though the device is already connected through "Adb Connect" and "Adb devices". I've already checked the "Always allow from this computer" checkbox, but sometimes a pop-up window appeared that failed the test and sometimes passes if not displayed. Can someone enlighten me how to tackle this problem?