Cesar Ramos started for the Rays a day after the team announced that he is has joined Jake Odorizzi, Erik Bedard, Enny Romero, Alex Colome and Nathan Karns as a candidate for the No. 5 spot in the rotation.
Ramos surrendered one-out singles to Eduardo Nunez and Ichiro Suzuki in the first inning, before former Rays Russ Canzler singled home Nunez. Kelly Johnson, another former Ray, grounded into a forceout to put the Yankees up 2-0 in the first.
Ramos allowed two runs on four hits while striking out two in 2 2/3 innings. Of the 42 pitches he threw, 27 were strikes.
"Felt pretty natural to be back out there [starting]," Ramos said. "The ups and down, I was more curious to see how I would react to it. I felt fine the whole way."
Grant Balfour pitched the fourth for the Rays, and the new closer surrendered a two-run single to Brett Gardner that put the Yankees up, 4-1.
Adam Warren, who is in the hunt for the Yanks' fifth-starter spot along with Michael Pineda and David Phelps, surrendered an opposite-field home run to Zobrist in the first.
Warren allowed one run on four hits, and he struck out two in 2 1/3 innings. He threw 44 pitches, 25 for strikes.
Yankees reliever Dellin Betances threw 1 2/3 innings of scoreless relief, but Jennings homered off New York righty Robert Coello with two aboard in the bottom of the fifth to tie the score at 4. Joyce added a solo homer to put the Rays up 5-4.
Up next:David Price will make his second start of the spring when the Rays travel to Sarasota, Fla., for a 1:05 p.m. ET contest against the Orioles on Thursday. The Rays' ace allowed one run against the Pirates 1 2/3 innings on Saturday. Also expected to see action are right-handers Heath Bell and Josh Lueke.