Domain backlinks influence on ranking?

I have noticed with most of the micro niches i'm finding that there's been a huge amount of domain backlinks, but not many page backlinks. Is it worthwhile competing against sites with huge amount of domain backlinks (normally high authority sites), or can you pretty much get in front of most of time, since domain backlinks are not really specific to the keyword. I would think it wouldn't have much influence since most of these backlinks are not relevant to the keyword, so it wouldn't be a big factor in ranking for that specific keyword.

Yes, it may affect ranking very much.

regards