Yes. First there is a general switch to turn it off. It is in the Little Snitch (LS) app configuration, but it is also on your top menu bar where you see the bandwidth usage barometer, for quick access.
However i rarely use it. When you first install LS, then again every time you install a new application that uses an internet connect, LS will ask what you want the rule to be for this particular application. For most, you should answer Any connection Forever. LS will remember the rule you choose.
If it is an application you don't trust, you can make the rule temporary, or deny it access at all. After you get use to it, LS is very unobtrusive most of the time.