Do I owe him?

I was playing pool against one of my mates. I was up for putting a bet on. Loser buys the drink. My mate didn't agree to the bet and wouldn't shake hands. After he won he said I owe him. I was like how - you refused the bet? He went home after as he needed to go. Do I owe him? If he had agreed to the bet I would have paid up.