Frost happens when the air temperature is near freezing, but not much below. Experts will tell you that it can't happen when the air is below freezing, because the moisture will already be frozen.
There are two types of carburetor frost, exterior and interior. The exterior frost happens when the cooling effect of the air drawn through the carb chills it enough below freezing that the moisture drawn from the surrounding air condenses on the surface and freezes there. The outside surface of the carb has to be somewhat below freezing for this to happen.
Interior frost can happen at a higher temperature than the exterior frost. This is because the air flowing through the venturi cools as it speeds up as it passes through. This draws heat from the air and the surrounding parts of the carb, cooling them below freezing and allowing the ice to build up, sometimes to the point of killing the engine when the carb quits working.
The intake manifold is designed to transfer some of the heat from the exhaust manifold to the carburetor to help combat this, but it can't overcome it all the time. The original cast iron carburetors help transfer this heat and hold it better than the pot metal kind.
Frost on the outside of the carburetor doesn't affect anything, but it may indicate that the venturi is being exposed to icing conditions.