The AC is off when the light on the AC button is unlit. The OFF you are refering to is just for the fan. I think if hte fan is off, the AC is automatically off as well, but I am not sure. To know the status of the AC, just look at the AC light, which is hard to see in bright conditions.
I don't understand why it has a light, yet doesn't function? even when "off" it still shows direction of air travel, why wouldn't it just be blank, and off? why does Toyota have to mess with me like this?