No-one knows what relevancy means exactly, so take all answers here with a pinch of salt.
My opinion (which may be wrong) is that Google looks at whether the page linking to you contains similar keywords (and references to entities) in its decision as to how much link juice to pass.
The topical relevancy of the majority of pages on the website linking to you (what you could call the domain of the website) is not so important, except in consideration of how much link juice is passed between those pages and the one that links to you.
If you read about Google, you'll see that most of the concepts it uses (such as those in its patents) are based on how individual pages are related to each other, not how a page is related to a domain name. A domain name is simply an address at which pages can be found. The content on the pages is what is important, not their address.
In other words, if your site is about cats and the page linking to you about cats, the link will transfer a lot of the link juice for that page. But the page itself won't have a lot of link juice transferred to it if every other internal link to it (from the domain its on) is from a page about nutritional supplements. So ideally you want a link from a topically relevant page that has links from other topically relevant pages (whether on the same website or on external websites).
