On my current project, we have been growing our system to a rather large size. As with any codebase that grows, this means that we have started to think about code organization that relies on creating our own libraries and gems.
At this time, we have around 5 or 6 "internal" gems that are just consumed by our project. These gems are things like third party integration classes, testing tools and generalized code that could be used outside of just this project.
One of the things that we've started doing is versioning and tagging the external gems instead of pulling from the master branch of the gems (we don't use a gem server -- just GitHub). This reason for doing this is really based around continuous delivery and the traceability of our changes.
A thought experiment: I give the QAs a build that they need to look over and verify. They find a regression. They report it back to the developers and you pick up the story to try and fix it. The first thing that you need to do is to reproduce the bug. How do you do that? Normally we would check out the commit that corresponds to the QA build and follow the steps. However, this bug is actually in the gem. When you bundle install that gem, you're pulling from the gem's master and not a specified tag. How do you get to the gem's specific version that actually created that issue? You'd have to do some time-based hackery with logs from the time that the original build was created. Ugly and error prone.
Instead, simply tag and version your gems and use those tags in your Gemfile. Then, when you checkout that specific version, all the information that you need about versioning is versioned right into your app. It'll make your life easier!