First, Wireguard is a UDP-based protocol, meaning there is no TCP three-way handshake involved. When someone wants to connect, they simply send their UDP packets. If they have valid keys, they will get a reply. Without valid keys, the packet gets dropped, and no error messages are sent—nothing.
This is one of the key security benefits of Wireguard: dropped packets don’t get logged by default, so your logs don’t get spammed with failed connection attempts.
What this means is that a hypothetical attacker has no real way to know if a Wireguard protocol is running on a given IP/Port unless they have a valid set of keys. The cryptographic keys used by Wireguard are so large that brute forcing is not a practical concern.
However, if you’re still paranoid about potential attacks, you could take extra measures by using nftables to create rules that log ‘NEW’ incoming UDP packets. Since connection tracking is required for NAT, the kernel can track whether a UDP packet has received a reply. You could even rate-limit the number of incoming ‘NEW’ packets, making brute forcing impossible, even if someone somehow knew you were running Wireguard and thought they could guess the keys.
Overall, using the default port (51820) for Wireguard is safe due to the combination of cryptography and how the protocol handles packet drops without sending replies to invalid requests.
There is one reason to use a different port than 51820, that is, to avoid port blockages which some ISPs or firewalls may implement for whatever reason. To do this, you could simply use 51821 or similar.
Additional reasons pointed out in comments:
To work around ISPs that use port-based traffic shaping / throttling. In those cases, often using a common UDP VoIP or gaming port can bypass this.
And if you have multiple separate WG server instances running behind the same public IP.