As the 'Rhythm and Soul' movement developed and evolved into Northern Soul, the term began to apply to anyone who was not into Soul music (ANY Soul music), and that is the way we used it in the early '70s as I remember, the expression being shortened to "Div"
Yep bang on this, thats exactly what the term was used for and always will be I guess, I still hear folks using it today, 37 years on(oh my God 37years) especially in some oldies venues. thing is most groups who follow particular styles, fashion, music whatever have a term to describe those not on the inside.
Div records imo:
Frank Wilson Do I Love You
Nosmo King Goodbye Nothing To Say
AL Wilson The Snake
and that dam awful Lost Summer Love,
Mr Bridget