A priority encoder in VHDL checks input bits starting from the highest priority bit down to the lowest. It outputs the binary index of the first bit set to '1'. If no bits are set, it outputs a default value, usually zero. The process stops checking bits once it finds the highest priority '1' bit. This behavior is shown in the execution table where inputs like "0101" result in output "10" because the second highest bit is set. When input is all zeros, the output defaults to "00". This logic helps hardware circuits decide which input has priority quickly.