It's probably best to take a step back and ask what you actually want an LLM to do for you. Using RAG and just having it raise references to consider is probably the best you can expect.
I have sometimes asked legal question (to which I pretty much know the answer) of LLMs and my consideration is that a lot of the time the section of an Act (UK) is the principle node about which a piece of information is anchored, but the anchoring is very loose and many sections are often mentioned in proximity, leading to poor differentiation of the relevance of a document/sentence/word vector to a particular piece of legislation. It might be fixed by training NER to recognise treaties/acts/conventions and the jurisdiction and always using that to label references; I suspect the "1" in "section 1" or "17 USC 1", say, is not being tokenised as part of the "section" and this contributes to poor results. Maybe someone has worked on this?
Also, context for the jurisdiction in which a discussion is taking place is often not really given - can the LLM tell that a law firm is talking about PCT, or EPC rather than USC when it discusses first-to-file for patent law and nothing in the document itself mentions the jurisdiction or any of those three initialisms? How about when the same law firm whose blog is mentioning these things represents the same clients at WIPO, EPO and USPTO? If you're going to fine-tune for that with human question-answer sessions you're going to need some really skilled workers who know the field well.
You probably then need specific prompt templates for legal questions too.
Then they need to layer in precedence and authority of different courts, recognise obiter statements from other statements, recognise summaries of positions of different parties aren't original statements by the speaker, disregard ultra vires statements, ... simples.
It's probably best to take a step back and ask what you actually want an LLM to do for you. Using RAG and just having it raise references to consider is probably the best you can expect.
I have sometimes asked legal question (to which I pretty much know the answer) of LLMs and my consideration is that a lot of the time the section of an Act (UK) is the principle node about which a piece of information is anchored, but the anchoring is very loose and many sections are often mentioned in proximity, leading to poor differentiation of the relevance of a document/sentence/word vector to a particular piece of legislation. It might be fixed by training NER to recognise treaties/acts/conventions and the jurisdiction and always using that to label references; I suspect the "1" in "section 1" or "17 USC 1", say, is not being tokenised as part of the "section" and this contributes to poor results. Maybe someone has worked on this?
Also, context for the jurisdiction in which a discussion is taking place is often not really given - can the LLM tell that a law firm is talking about PCT, or EPC rather than USC when it discusses first-to-file for patent law and nothing in the document itself mentions the jurisdiction or any of those three initialisms? How about when the same law firm whose blog is mentioning these things represents the same clients at WIPO, EPO and USPTO? If you're going to fine-tune for that with human question-answer sessions you're going to need some really skilled workers who know the field well.
You probably then need specific prompt templates for legal questions too.
Then they need to layer in precedence and authority of different courts, recognise obiter statements from other statements, recognise summaries of positions of different parties aren't original statements by the speaker, disregard ultra vires statements, ... simples.