Yes, but the claim is about "unlimited context length." I doubt attention over each segment can be as good at recall as attention over the full input context.