mgcghst
There must be another dialog from adb asking you if you wanna trust the device. Sometimes it does not appear/is hidden behind the dialog you described for the file access.
If this happens it is best to start a terminal and execute adb devices, which should trigger the dialog to trust adb from this device again.