Can't help you with Outlook. If you can make birthdays show up there it will be a setting in Outlook, and it will only work for contacts in Outlook.
But for iCloud.com, on the site with the Calendar open, click on the gear icon at the top right and choose Preferences. There is a checkbox in the Appearances section to enable the Birthdays calendar. Takes a few seconds to load once you save the Preferences.