X Server terminated Error ~ Connection Refused

Are you still getting this if you do sudo clr-boot-manage update? It sounds like this issue: EFI Errors after update/repair · Issue #208 · clearlinux/clr-boot-manager · GitHub and will not update your kernel

For future reference the nvidia installer drops the file in /usr/lib/modprobe.d/nvidia.conf. If you followed the documentation or @doct0rHu’s scripts this file doesn’t get added because we use the --no-nvidia-modprobe flag on the installer command line.

Try this: CUDA Toolkit Installation Error - #9 by puneetse