Without measuring this it is difficult to say. Polarity is also a function of charge separation. T-butyl is a better electron donor than sec-butyl. This could result in a higher negative charge on the chlorine and a higher positive charge on the teritiary carbon of the t-butyl group. This charge separation could cause the t-butylchloride to be of higher polarity.
The permittivity for sec-butyl chloride is 8.564 and that of t-butyl chloride (2-chloro-2-methylpropane) is 9.663. (CRC Handbook of Chemistry and Physics, 84th ed., Chap. 6, p 161.) That of 1-chlorobutane is 7.276. It seems to follow the carbocation-stabilizing ability of the carbon attached to the chlorine. The permittivity for 1-chloro-2-methylpropane, another form of butane that has a primary chloro group, is 7.027. This is somewhat lower than 1-chlorobutane (7.276) which indicates that the permitivitty is not solely determined by the nature (primary, secondary, tertiary...) of the chlorine substitution.