Don't immediately discard streams with 0 references