You can, and that is effectively doing something similar to putting a style tag in the head of your document. It'll block until the styles are parsed and applied. If you have a ton of styles, your Lighthouse score will suffer because your first contentful paint will take longer.
Well yeah if you want to avoid a flash of unstyled content then you have to wait for the browser to style the content. The main problem is the style duplication issue which someone else pointed out elsewhere in this thread.
The browser cache doesn't help on the first visit when the cache is empty, and the cache will also have to be invalidated whenever the component's CSS changes. And if the CSS file isn't cached then you also have to contend with the flash of unstyled content problem while the browser loads the CSS file. The Lit docs do a good job of explaining this: https://lit.dev/docs/components/styles/#external-stylesheet
