You might call it "ad hominem" as it is an attempt to discredit Diffie, although that usually describes a logical fallacy - the personal attack doesn't affect the validity of the argument. In this context the main reason for having Diffie there is because of his status and achievements, so it's a reasonable counter to try to discredit him.
I would say that kind of attack is not very fair, and a little bit dishonest, but not a flaw of reasoning.
It insinuates that the witness is unqualified to make the statements he's making, using logic of the form "if you don't have a master's degree you likely can't be an expert in this field."
I would have loved to response to ""You never completed a master's degree, correct?" he asked Diffie." with "Just like Bill Gates and Steve Jobs, no I don't have a masters degree" (well apparently Bill eventually got one...). I assume the judge would be unhappy with this, which strikes me as unfair - surely I can defend my reputation?
Note the very important difference between presenting a logical argument and serving as a witness for trial. When you present a logical argument, it stands on its own. But a witness isn't presenting reasoning for the jury to follow. His testimony doesn't stand on its own. A witness expresses an opinion or recollection of events for the jury to believe. The jury's job is to determine which witness they find more credible.
Facts that are ad hominem in the world of logical argument are vital determinants of credibility in the world of witness testimony. They help juries figure out who to believe. Thus the logic isn't "if you don't have a master's degree you likely can't be an expert in this field" but rather "the guy with the PhD is more likely to be a credible expert in this field." Which is a reasonable presumption. It's the job of the side presenting an expert with unique experience who doesn't have marks of academic qualification to convince the jury of the person's expertise regardless of the traditional indicators.
I'd be tempted to use the person within the organisation who's done lots of published research even without the diploma. But you make a good point - trials are about persuading the jury.
As a comment posted to another participant before my comment here to you pointed out, attacking a witness's statements on the basis that the witness may be biased, or may not have a basis of knowledge for what the witness said, is all standard procedure in a court case. The rules of trials allow the finder of fact to be informed of why not to believe everything that is said during the trial. It's up to each side in the litigation to follow the rules of evidence (a THICK book, interpreted by thousands of court cases) to suggest why some evidence presented at trial is better than other evidence and more useful for truth-seeking.
The adversarial system is designed to allow these kind of attacks.
"You don't know what you're talking about" is easily countered by "I invented these crypto systems; these have had no serious challenges in the X years since invented; I've defeated these crypto systems; I've published these papers in these peer reviewed journals; I have these honorary awards from these universities." etc etc.
Goes to the credibility of the witness your honour...
Reality is that the jury are only going to have a relatively limited understanding of the detail of these things so they're relying on expert witnesses. Part of that is whether that person has any right to call themselves an expert in the first place.
I am surprised he was allowed to get away with it.