Bushwalking/Hiking and Birding

I went for a buswalk today in the lower Blue Mountains (Florabella Pass) earlier today. Didn't really see any birds, as it was probably too hot for them (they were smart enough to find a patch of shade and stay there!), but I brought my birding lens with me anyway. I was getting annoyed by having to drop my pack and rummage every time I wanted to photograph a bird (which usually flew while I was stuffing around). So, I was wondering if anyone had a better system they used than this, or is it best just to divide time into Bushwalking and birding? Or would I be better off keeping my birding lens on my camera and switching to take other stuff?

I'm sort of dreading that the solution will be to buy some cheapo carabiners and simply hang my lens off my pack where it is easily accessable. 

Which leads me to part 2: I am hoping to do some overnight hiking next year, so I was wondering if it would be better just to leave my birding stuff at home for those, because the two do seem incompatible. 

Thanks for any help!