What makes a good advertising agency?

Make sure you know who will be handling your account and what their track record is. Examine their experience with your type of business and their understanding of your competitors. Make sure you know how they measure campaign effectiveness. Ask them about their fees and what's included - for example, whether they accept results-based payment. It would be helpful if you could get references from their clients.