Not sure if this complicates things, adds info or is completely irrelevant to the discussion, but may as well throw it in the pot. 'The Charms' 45 on Emblem (as in "If You Got the Notion") were in fact Moses Dillard and pals on most of the instrumentation (he arranged IYGTN, and the band did studio backing for a number of artists at Mark V in Greenville, SC). Johnny Few however was definitely lead singer on IYGTN.
The Charms 'Unlimited' "What goes up must come down" 45 on the other hand was recorded at Arthur Smiths studio in Charlotte, NC - Don Strawn (named on credits) was that studio's main sound engineer.
Meanwhile, me and Nat Speir of the Rivieras, outside the entrance of The Cellar in Charlotte...
If nobody gets back to you with a more direct contact, Michael Gray, editor at the Country Music Hall of Fame and Museum in Nashville can probably point you in the right direction. He has his foot in both camps, working for a country music organisation obviously but also a lover of local R&B / soul. Authors Colin Prescott and Peter Guralnik may also be approachable? I'd imagine they'll have connections given their history and through the recent book on Sun they worked on.
Songwriter / producer Bob Johnson might well have written, or at least co-written, this one. In fact, he could have written a number of songs which were credited to his wife Joy Byers, including some on Elvis, Timi Yuro and others. Bob was freelancing for Dot in the early 60s, and the Rodge Martin version was first. Problem is Joy was a songwriter too. Bob and Joy are both no longer with us now so relative contributions will always remain a mystery. I tried to clarify things with Mac Gayden a while back but these Nashville cats stick together!
As an aside, you listen to other Byers-credited compositions like Freddie North / Jimmy Church / BB King's The Hurt, or Rodge Martin or Susan Rewis' They Say etc, or Timi Yuro's What's a Matter Baby, or Elvis' Let Yourself Go etc. there are structural similarities to greater or lesser degrees. All great songwriting regardless.
Another random bit of info - I'm pals with Nat Speir, founder of the (Carolina) Rivieras - around 1963/64 he was involved in a series of sessions in New York to play on some of the Maurice Williams and the Zodiacs recordings for Herald. As it's the same time and label as the Jimmy Parker, I'll ask if he remembers him at all. The Jimmy Parker version is best left alone of course, but in the scheme of things it is at least historically relevant.
Bob Johnson also ended up head of Columbia's Nashville operations, so no surprise the song was recycled (yet again) for Susan Rewis.
Windlesoul's post in What Goes up Must Come Down (Soul Inc / The Charms Unlimited was marked as the answer