When you watch the logs before creating a particular protocol, it'll call denied connection attempts "Unidentified IP Traffic" or something like that. If you then create a protocol that encompasses those connections, the logger will refer to your protocol name. This illustrates how ISA Server "cares" about your protocol definitions whether you use them or not.
I've wondered, what if you make multiple protocol definitions that use some of the same primary ports? What would the logger call the traffic? What rule would it know to obey if there are logical collisions like this?
I think the act of you defining a specific protocol encompassing multiple ports as a primary connection sets a certain boundary. So if activity falls within that realm, it is assumed to be communication of that protocol, and since you've disallowed UDP, it gets dropped. Maybe if you made the UDP definition part of the secondary protocol def there would be different behavior, because the udp traffic wouldn't be "key" to the protocol?
As you mentioned, our discussion is academic because for Steam you should probably be using Send-Receive for UDP.
But my guess as to what the deal is, is that you have some overlapping protocol definitions. If one protocol definition has the same primary connections as another, how does ISA server know which protocol your IP traffic falls under?
Anyway, for the record, this is my Steam definition. I haven't tested it with online play yet, but it works for getting updates from steam, etc, and is built based on the protocol info in their own forums:
You could consider moving the UDP to secondary if this issue ever messes with you in the future. I don't know a lot about sockets programming for games, but I would imagine that the UDP information flows only after a TCP session is established, and therefor it would be safe to make it secondary. This, I think, would give ISA Server's ability to observe "statefulness" the means to logically separate the steam protocol from other protocols you use that also use similar UDP traffic.