Look for designated zone signs. Your permit is only for your zone.

Allentown Parking Authority's Parking Zone Map