AHH! How to keep bees from coming in the fireplace?

We rented a house in January and with the warmer months there have been some bees, wasps, hornets, ... all of it coming in from our fireplace! I'm allergic and it's just scary to have them coming from there - how can we keep them from getting in.