According to my dealer, the pollen filter feature filters the air that is already inside the cabin. Don't the other cabin filters in other cars only filter the air that is entering the cabin from the outside? Pretty sure that's the case, as when replacing these in my other cars, they have contained pollen, pine needles, etc....pretty sure that what's different about this and might not be in your Prius.The manual from the online Lexus service says:
"Micro dust and pollen filter button.
Operates micro dust and pollen filter on/off. Outside air mode switches to (recirculated air) mode. Pollen is removed from the air and the air flows to the upper part of the body. Usually the system will turn off automatically after approximately 3 minutes."
But I'm not sure why you would want to bypass it. I know our Prius always filters the air.