By 1962, Ray Charles had fully crossed over. It started with his 1959 Top 10 hit “What’d I Say,” continued with the Grammy-winning “Georgia On My Mind” and “Hit the Road, Jack,” and culminated in two smash country music albums, yielding the number one single “I Can’t Stop Loving You.” Proficient at bebop, fluent in country, and having practically invented soul, it seemed there was no style of popular music Ray Charles couldn’t master. Beyond his prodigious songwriting and piano playing abilities, Charles was most famously a vocal interpreter. With his newfound wealth, he founded Tangerine Records in 1962. The first thing he did was produce and release a record by one of his favorite singers, Jimmy Scott.
Jimmy Scott had been recording since the late 1940s and made several notable if unprofitable albums with Savoy Records in the 1950s. His work was almost universally loved by the most influential vocalists of the era, a group that included Dinah Washington, Ruth Brown, and Billie Holiday. Ray Charles implicitly understood the singer’s potential and believed he had the key to Scott’s elusive success.
“My concept was romance,” Charles told Scott’s biographer David Ritz. “Make a romantic record you could listen to late at night with your lady. I wanted the kind of record you could play over and over again, where you wouldn’t be bored and the mood stayed steady.”
The result was Falling In Love Is Wonderful, released in June 1963. It is a seminal album in Jimmy Scott’s discography, a triumph of performance, production, and arrangement. His languid, plaintive voice is surrounded by a lush orchestra and supported by Charles’s jazz-and-blues-tinged piano. Half a century later, it is rightly considered Scott’s masterpiece.
But as a result of contractual disputes with Savoy Records and its aggressive owner Herman Lubinsky, Falling In Love Is Wonderful was pulled from stores and shelved about a month after its release. It remained unissued for almost 40 years. A subsequent record with Atlantic suffered the same fate. The accruing emotional weight led the singer to completely withdraw from the music industry until later in life.
Kickstart your weekend reading by getting the week’s best Longreads delivered to your inbox every Friday afternoon.
Like so many things, the apparent story of Jimmy Scott is at odds with the actual story: Had it stayed available, Falling In Love Is Wonderful might not have proved the career-launching album it’s claimed to be. He lived in a culture that needed to catch up with his singular gifts, and we’re still in the process of achieving that kind of acceptance.
Born in Cleveland, Ohio, in 1925, James Victor Scott began performing at an early age. In the early 1940s, he talked his way onto a bandstand in Pennsylvania, and was soon fronting for the likes of Lester Young and Erskine Hawkins. In 1948, vibraphonist Lionel Hampton hired him on the spot after a performance, calling the diminutive and skinny 23-year-old “Little Jimmy Scott.” It was a name that would stick, because Scott never reached sexual maturity. A genetic disorder called Kallmann syndrome prevented him from completing puberty. As a result, Jimmy’s voice never dropped. He was a natural castrato, a permanent contralto, as evidenced by 1955’s “When Did You Leave Heaven.” His voice was genderless.
“You had to realize one thing,” Scott said years later. “I felt, ‘Hey, I’m as much man as anybody else.’ And my mother would say, ‘You’re a man,’ and make you think the right way about being a young man.” Personally, he was subject to constant harassment. Professionally, his condition was a double-edged sword. Audiences and most vocalists loved him; others were more interested in taking advantage of his talent than giving him his due.
Scott sang on the Top 10 R&B hit “Everybody’s Somebody’s Fool” with Lionel Hampton’s band in 1950, but — typically for the time — went uncredited. (His vocal on a live recording of “Embraceable You” with Charlie Parker that same year was credited to female singer Chubby Newsome.) After a fight over money, Scott left Hampton’s band and returned to Cleveland, where he worked in a hotel and sang in small clubs. He signed with Roost Records, who at least put his name on the singles.
A subsequent move to Newark, New Jersey, put Scott in the path of Herman Lubinsky, head of Savoy Records, the label of Wynonie Harris and Charlie Parker. In an era of exploitative contracts, terrible royalty deals, and the utter venality of label bosses, Lubinsky distinguished himself. He was, according to Atlantic Records executive Joel Dorn, “a hemorrhoid of a human.”
“There is no doubt that everybody hated Herman Lubinsky,” music writer Tiny Prince remembered in Swing City: Newark Nightlife 1925–50. “If he messed with you, you were messed. At the same time, some of those people — many of Newark’s top singers and musicians — would never have been exposed to records if he didn’t do what he did.”
Scott signed a terrible contract with Savoy Records (almost certainly making less than 5 percent royalties) without any legal guidance. The material he was subsequently given to record — a passel of mostly mediocre R&B songs — was not to his liking. “You go through it,” he remembered about his relationship with Lubinsky. “You go through it, because he throws in your mind about he owns you and the contract he’s got on you. You don’t know how they negotiate those contracts and things. You don’t know, because I never did know just what it was, and when I found out, it was too late.”
Still, the records Scott made for Savoy illustrate his evolution from an R&B crooner to a masterful balladeer. By the time Ray Charles came into the picture, Scott had graduated from singing tentatively over vibraphone and guitar-led small bands, to giving full-throated performances, backed by strings. By the end of the 1950s, his voice achieved an extraordinary power and authority.
Scott had a boy’s range but a man’s diaphragm, giving him operatic projection. He could fill a room without amplification. Moreover, unlike most male singers, Scott never sang falsetto. Every note, even in the top of his range, was subject to the same power and control. His throat muscles developed to produce an impressive vibrato, which Scott used to communicate intense, but not melodramatic, feeling. By his mid-30s, the singer despaired of a deeper voice, even though a growth spurt gave him an additional eight inches of height. “Well, I learned that it was a gift that I was able to sing this way,” he told the New York Times. “Many times, I’d think, ‘I’d love to try this in a lower register’ … but then after a while you think, ‘Sing with what you got.’”
By the early 1960s, at the height of his vocal power, he grew tired of the penury and restrictiveness of his label and broke with Savoy. His girlfriend at the time, a backup singer in Ray Charles’s band, convinced him to move to Los Angeles. Brother Ray wanted to make a proper Jimmy Scott album.
“Jimmy had a tear in his voice,” Charles once said. “It brought out a sadness you couldn’t help but love. Great ballad singers are rare. Anyone can mouth the words, but how many can make you believe the story? Jimmy made a believer out of me the first time I heard him.”
For Scott, this all was a dream come true: Los Angeles was a land of career opportunity, and Ray Charles was one of the most successful, savvy musicians around. Perhaps best of all, Scott was allowed to choose his own material. “And we were beginning to make a list, he and I,” Scott remembered. “I picked tunes. He said, ‘I know that.’ He said, ‘That’s beautiful, man.’ ‘OK, we’re going to do that.’ Write it down.”
“Ray supervised,” arranger Gerald Wilson said of the session, “but he did so from the piano. He played on every track. And he made sure we wrote charts that gave his piano space. If you listen to the record carefully, it’s really a long and intimate conversation between Ray’s sensitive piano and Jimmy’s sensitive voice.”
“That was beautiful,” Scott remembered. “I felt like a king. … Ray Charles is playing for me.”
The album was cut quickly, and mostly live. “The record was completed in just a few sessions,” Scott told his biographer. “I don’t think we did more than two takes on any one tune. There wasn’t any overdubbing either. It was all-the-way live. The fiddlers were fiddling. Ray was playing, and I was singing, all at the same time. When we were through, everyone was thrilled — me, Ray, and the arrangers. We knew we had a hit.”
Falling In Love Is Wonderful can rightly be compared to previous highly orchestrated masterworks: Frank Sinatra’s In the Wee Small Hours, Billie Holiday’s Lady In Satin, The Genius of Ray Charles. But thematically, Scott could go where those others could not. His voice communicated transcendence more than ambiguity. Frank Sinatra and Ray Charles’s musical identities were heavily dependent on being men; Billie Holiday sang from a resolutely female perspective, with much of her best-known material concerned with wayward, abusive male lovers. Scott elided this binary. It’s no accident that all but one of the ten songs on Falling In Love Is Wonderful have no gender-specific pronouns in the lyric. Only “Someone To Watch Over Me” changes the original male object of desire to female, but even this underscores what, for the time, must have been an emasculating vulnerability. “I’m a little lamb who’s lost in the wood,” Scott sings.
I could be good
Oh yes I would
To someone who’d watch over me
“I Didn’t Know What Time It Was” simply leaves out the first verse, written about male paramours: “Once I was young / Yesterday, perhaps / Danced with Jim and Paul / And kissed some other chaps.”
Because he chose the material, this was the way Jimmy Scott wanted to express himself, and it was the best vehicle for his expression. Romance is most authentic when two loving people meet as equals, when identity is free from cultural expectations of dominance or submission. In a way, Scott also guaranteed his limited success: Not only did artists like Sinatra and Charles trade on their robust sexuality, but their masculinity was also an easy reflection of the male listener’s own ego identification. Most men were put off by Scott’s androgynous appearance and voice, because it threatened their own sense of self and sexuality. Women and queer people were more devoted fans: Scott admitted to carrying a gun to ward off homosexual advances. (“They would try you,” he told The New York Times. “Lots of times, I was tried.”) One can’t help but wonder if Scott’s reaction to unwanted advances may have stemmed from his frustrations at not being obviously heterosexual in a deeply homophobic culture; in any case, his hostility receded with age.
The music industry could not countenance Scott. The cover of Falling In Love Is Wonderful did not feature the artist at all but instead had a picture of an anonymous man and woman on the floor in front of a fireplace. The man’s tie is undone, and the woman lies prostrate beneath him, looking out of frame, a picture of surrender. Ray Charles records are scattered around them.
“I was disappointed not to be on the cover, but I understood,” Scott told his biographer. “The important thing was the music — and the music was everything I’d dreamed of.”
Joel Dorn, the future Atlantic Records executive, was still a DJ in Philadelphia when Tangerine released the album. “When I put that Tangerine record on, the phones just lit up,” he remembered. “Everybody was asking, ‘What’s her name?’”
“In the six years I was on the air,” Dorn said, “I never had a response to a record like that. We had a phone in the studio and people could call us up while we were on the air. No record elicited a response like that Jimmy Scott record elicited.”
Jimmy Scott was 37 in 1963, and he still looked like a boy. As an artist, he was neither fish nor fowl: An orchestral album of standards was probably not going to burn up the charts. Although a putative jazz singer, Scott’s deliberate, theatrical style had fallen out of favor by the time bebop came around. Rock ‘n’ roll blazed a commercial path years before, torching R&B and jazz as profitable expressions in its wake. Even Falling In Love Is Wonderful’s liner notes tried to cover a very wide demographic spread: “Just in case the pressures of the business world have made you forget,” they began, “or the passing years have dimmed your memory, or you are not yet old enough to really know, we want to remind you that Falling in Love is Wonderful.”
Immediately after the album was released, Herman Lubinsky contacted Charles. He claimed that Scott had signed a lifetime deal with Savoy and was still under contract. He was bullying and insistent. “He told Ray — he even threatened the boy,” Scott said. “He said, ‘I’ll have you put in jail for life.’ He said, ‘Because that’s my artist that you recorded.’ He kept threatening, calling him, threatening him.”
The Father of Soul capitulated. “Tangerine was new, and I didn’t want to go through the expense of a legal fight,” Charles said. “So I pulled it.” Scott couldn’t do anything about it either. “I was broke,” he admitted. “And besides, I didn’t know any music lawyers. In those days, who did?”
And so, barely a month in the stores, Falling In Love Is Wonderful was shelved. It became an object of desire for collectors and fans, its value lying in scarcity for the former and quality for the latter. “Unearthing a copy of Jimmy Scott’s Falling in Love Is Wonderful was tantamount to finding a signed first edition of The Catcher In the Rye or unearthing a ’69 Yenko Camaro,” noted Jazz Times.
Joel Dorn did not forget about Jimmy Scott. “I got to Atlantic and had a few hits under my belt,” he told an interviewer. “I signed Jimmy. … I was under the impression, and so were Atlantic’s lawyers, that Jimmy was free and clear of Savoy, so I went and made an album with him. Out of the woodwork comes Savoy and boom! The album’s taken off the market in like three weeks after it’s released.” That album was called The Source, recorded in 1969. Instead of a picture of Jimmy Scott, its cover featured an attractive young black woman. It is of the same quality, in production, songs, and arrangements, as Falling In Love Is Wonderful.
Despite his cartoonishly villainous persona, it’s fairly clear that Lubinsky wasn’t simply working to keep Jimmy Scott down. Rather, he was brazen enough to try to enforce a “lifetime” contract, and canny enough to know that other labels wouldn’t go out of pocket to call his bluff. He would have been willing, it seems, to lease Scott out to another label for some consideration. He had already done the same thing years before for King Records, though that label didn’t have nearly the same kind of clout and distribution power available to Tangerine.
“He felt like he put good money into Jimmy with little results,” Herman Lubinsky Jr. said of his father. “When Jimmy suddenly turned up with an album distributed by ABC, Dad was incensed. If Jimmy had initially come to [my father] and asked for permission … I have a feeling that my father would have agreed.”
But Lubinsky’s notorious reputation clearly preceded him, and neither Ray Charles nor Atlantic Records were interested in entering into any kind of agreement with the man. Even in an industry known for rapaciousness, entering into a contract with Savoy would mean making a deal with the devil.
Jimmy Scott moved back to Cleveland, becoming, by turns, a retirement home nurse’s aid, a short-order cook, and a hotel clerk. In 1984, a friend (who Scott would go on to marry) called a Newark radio station and requested one of his songs. The DJ told her Scott was dead.
Very few people fought for Jimmy Scott. Not Lionel Hampton, who refused to meaningfully pay the singer who gave him a hit; not Herman Lubinsky; not Atlantic Records, who chose to take a loss on recording costs rather than fight for an artist they believed in; and not even Ray Charles, who, in the same year that The Source was pulled from shelves, replaced Scott’s vocals on Falling In Love Is Wonderful with Wild Bill Davis’s organ, releasing the result as Wonderful World of Love.
Scott remained the only singer Charles ever produced. “Jimmy was singing soul way back before the word was being used,” he once said.
Scott did have one champion, the great Doc Pomus, a songwriter responsible for “A Teenager In Love,” “This Magic Moment,” and “Save the Last Dance For Me,” among many other iconic hits. In 1987, Pomus wrote a letter about his friend to Billboard. “If you want to know more about his singing,” Pomus wrote, “ask Quincy Jones or Stevie Wonder or Frankie Valli or Nancy Wilson — they idolize him.” He went on:
When we talk about Jimmy Scott we’re talking about somebody who might be the best singer of contemporary or vintage ballads around. There must be some space somewhere for him. What’s everyone waiting for? He’s 62 years old, he’ll die and there’ll be a hot funeral. Everybody will show up in hip mourning clothes and talk about how great he was. Let’s do something now.
It was when Jimmy Scott sang “Someone To Watch Over Me” at Doc Pomus’s 1991 funeral that he was rediscovered, given a five-record deal, and allowed to have a proper, if third act, career. David Lynch used him to great effect in episode 29 of Twin Peaks, singing the haunting “Sycamore Trees.”
I got idea man
You take me for a walk
Under the sycamore trees
The dark trees that blow baby
In the dark trees that blow
And I’ll see you
And you’ll see me
And I’ll see you in the branches that blow
In the breeze
I’ll see you in the trees
I’ll see you in the trees
Under the sycamore trees
Here is Jimmy Scott put to luminous use again, three decades after Falling In Love Is Wonderful. He’s free from his confining and transient body, free to be intimate without the neediness of desire, free to be perceived as not quite man and not quite woman. You could call his voice angelic, were it not so informed with yearning and loss.
The Rhino Handmade label released a limited edition of 7,500 copies of Falling In Love Is Wonderful in 2003. (Although these sound like — and are — modest numbers, the digital age makes a critical difference: Sound files are not a physical product and are thus infinitely reproducible. We will always be able to listen to Falling In Love Is Wonderful even if we don’t own the album.) Jazz critics lined up to bestow praise: Michael Bailey, writing for All About Jazz, declared Scott’s ethereal voice “a national treasure.”
“I appreciate the fact that these things are finally happening for me,” Scott told the Cleveland Plain Dealer in 1997, “but I wish they could have happened earlier in my career so I could have enjoyed the retiring years much better.” Upon reflection, he added, “in show business, generally you don’t retire. If you love it, that is, you’re in it forever anyway.”
***
Tom Maxwell is a writer and musician. He likes how one informs the other.
Editor: Aaron Gilbreath; Fact-checker: Ethan Chiel
Get the Longreads Weekly Email