The Evil GM - Background Scenery - Do players care?

Was sitting down last night with all my books out on the table, planning for my D&D game.. I was at the part where I was thinking about designing up the town the PCs will be spending some time, How much time that was, depended on them. 

We all know you plan for things a certain way and either, its never used or its barely touched upon in the game. Good news is maybe your efforts can still be salvaged in a later attempt as you can always use in another session or two.

Do players really care, if you have set up the background, such putting in colorful NPCs, give details such as you hear the wind-chimes of the general store, while every once in a while you get the smell of a wet dog as you pass the watering hole area through your nose.

Sure that is all set up and mood setting, but should you continue to make the background colorful and alive, or just say MEH! moving on to the next thing as they really don't care?

I know I enjoy designing up the background stuff just as much as planning the entire adventure for the night. 

Its fun to see a town come to life on paper, and its even more fun to play it all out, but... 

Sometimes I get the feeling players don't give a crap.

Sure, I know, it all depends on the players and their personalities... blah blah blah. Yeah, I've heard it before and I know this.

What do you think?