I think there is an easier way in Illustrator, but I use photoshop and to achieve a dotted line inside a circle say, I will duplicate the circle layer and shrink it down to where I want the dotted line to be. While you still have the shapes path visible, goto the paths section and double click the shape mask, give it a name and hit enter: this will make a path you can create your dotted line with. Now you make a New layer, goto your brush tool and make sure brush options box is open (F5), turn off scattering and on the brush dynamics increase the spacing until you get the desired dotted line. Once the brush options are set, make sure you have the new layer selected, go to your paths dialog box, select the circle path u created, and click the option "Stroke to Path" (far left icon at the bottom of the box). This will make a stroke with the brush you chose following the path.
I have a feeling this was a very complicated way of explaining it , so if you have any questions feel free to ask!
Also, it helps if you show us an example of an image that has the effect you're looking for.