When categorizing custom post types, why would categories follow over?

I'm after shooting myself in the foot here somewhere; I've added a list of parent and sub-categories to a custom post-type,

The second post type doesn't need categorization - but the categories do follow over, why would this happen do we know?

I'm asking as I deleted the categories from the second post type, and it removed all the categories from the post type that needed them...

I've just had the idea to export as JSON and see what overlaps, I'd be really grateful for any responses with some technical wisdomz.

Does added categorization to places with no use for categories bloat anything the big G wouldn't like bloated?