- Arti Rai, I'm Co-director of the
Center for Innovation Policy at Duke Law,
along with Sarah Benjamin and our executive director,
Steve Merrill.
So, this is the first use case.
We're gonna have three use cases during the day
and in some respects, I think this is the best use case,
not just because I'm in intellectual property lawyer,
but also because at least in the patent world
and the trademark world, it's all about pragmatism.
It's all about improving economic growth.
And so, we don't have to worry about nondiscrimination,
I don't think,
Although, maybe you guys will come up
with a clever way to worry about nondiscrimination.
We don't have to worry too much about justification,
although, so these, I'm going down the concerns
that were expressed in the last panel.
We don't have to worry too much about justification.
Although perhaps, we can get into that as well.
But we do have an extreme problem and that is the problem
that the Patent Office receives
about 600,000 patent applications every year.
And it's got a finite number of examiners to review
those patent applications to search the world's knowledge.
I know, Google, you're suppose to search the world's
knowledge, but actually, the Patent Office is suppose
to do that as well.
Search and classify the world's knowledge,
that's part of the mission of the U.S. PTO.
And it has to do that with human beings right now
and a limited number of human beings
'cause it's always budget constraint.
So, wouldn't AI be the perfect thing
to use in this situation,
search the world's to classify the world's knowledge?
For addressing that subject, we have my dream panel,
absolute dream panel, Liat Belinson,
who is the CEO of AI Patents, guess who she is.
(laughing)
Ian Wetherbee, who directs Google patents.
Google is an incredible, I have been the beneficiary
of Google's incredible font, the huge amount of information,
huge reservoir of information they've put out there
on patents and I use your data all the time.
So, we have Ian Wetherbee from Google
and he can tell us about how Google thinks about
questions or searching the world's knowledge
and classifying the world's knowledge.
Scott Beliveau from the US Patent and Trademark Office,
who is and here I will read from his bio
because he's got such an impressive title that,
and I always forget it.
He's the Director of Enterprise Architecture
and leads the Advanced Enterprise Analytics branch
at the PTO.
Very impressive, even though he looks like he's in his 20s.
(laughing)
And then finally, we have Alex Measure,
from the Bureau of Labor Statistics and you might wonder,
well, that's not exactly IP and patents,
but it is administrative data of a sort that I think
will give some interesting inter-agency perspective
to what the others are gonna talk about
with respect to patents.
So it'll be mostly patents, although maybe Scott, you can
say a little bit about trademark as well.
So, without further adieu, each of the speakers
is gonna speak for about seven to ten minutes.
I will have perhaps a question for each speaker
after they speak, and then we will open it up
to general discussion.
Yeah.
- Thank you.
So first of all, thank you very much for giving us
an opportunity to take part in such an important event.
I'm the founder and CEO of AI Patents.
Basically, the motivation behind founding this company
is that every idea can be described in multiple ways.
I can call a cup a container that holds liquid.
Patent documents as you all know are
50,000 words in average.
You have about 40 million documents only within
the U.S. PTO, which has continued to grow every year.
The hard path that inventors, patent practitioners,
and examiners have to do is to assess quality patents
(mumbles) search, against each application that gets
into the U.S. PTO.
But to run against all those 40 million documents at least.
The big problem that we identified is basically
how do you represent these documents if you only key words?
How can you just (mumbles) and the method to
identify prior art?
This is a system that will allow users to just
freely describe using their own words.
Copy, paste, claim, description, or even broad description
into the search engine can benefit the searcher
in the search process to be more comprehensive
and to find more relevant information
when they conduct a search.
So, the approach that we've done basically kind of
has three parts.
The first one is what is a document and what is a (mumbles)?
Unlike other approaches, we didn't take only
the top 20 key words.
Or we didn't (mumbles) for that or citation analysis.
But what we've done is basically took the whole document.
There is a reason why the patent is so low.
It contains description from the inventor.
It contains a legal and description from the
patent attorney.
And all of this is (mumbles) in order to conduct
a pure search.
So, basically what we're doing, we really take the raw data
and index, it can make it searchable.
The second part, which is kind of on top of it
is the AI part.
We have a (mumbles) on basically learning from the expert.
The experts are the examiners.
It is a cumulative knowledge of years and years
that identify that these two patents shares the
same year even though the text is different.
And what we've done is basically use this information
through 102, 103 rejections in order to first
construct a dictionary.
This dictionary is dynamic unlike other language
dictionaries because it keeps learning from the
experts as we go.
And it's dynamic because technology involves and creates
new terminologies being introduced.
And it's important that our tools will also be part of it.
So, this dictionary is basically, you're able to get
from documents level to the term level.
So, for ever technical term, we present a user,
suggested other associated terms that he could take
into account when he conducts a search.
The last part of this is basically okay.
We run the search, we have (mumbles) document,
if they're in a backend like they are in today,
you have to go through thousands of thousands of documents
which is a very tough thing to do.
So, what you've done is stack the results,
sort it by relevancy, in which we also incorporated
the examiner's decision of how this (mumbles)
translates into decision making in terms of novel.
And this is basically you can see in the middle,
you basically present results sorted by relevancy
which helps the user to get more information
more comprehensive, less restrictive in order to go
to the next step, which is key word to Boolean
which we definitely realize that this is an
important part of the search process.
So, it doesn't come to replace the expert,
and doesn't replace the current method.
It comes to compliment it in order to do a better job,
a more comprehensive, and to add more quality
to overall patent system.
This can be obviously iterated in order to shape.
Now, in terms of usages, we have companies
that are using our search engine as a prior art search
and due diligent purposes.
So, kind of assess whether (mumbles) such technology.
There's licensing, implication on that as well.
We were subcontracted by DOD in order to create measure
for novelty of patents.
And again, the search results we kind of represent
is a vector of which documents are closed and (mumbles)
which is basically an input for how novel it is
or how (mumbles) the patents are.
We also obviously had the honor to work (mumbles)
with the newest PTO.
The idea is we were selected out of 22 companies
and the goal was to really examiners in the search process
by presenting them a search report before they
actually conduct their own search.
Additional implication is basically classification.
Using the vectors I just claimed, long text documents.
We are able to see that you could find closed patents.
Even though they're actually attributed to different IPCs,
we identify them as one.
So, this is another example why taking the whole text
of the document is really important also
for specification approaches.
So, in terms of vision, I think that we all get (mumbles),
we need some kind of AI automated tools that will assist us
in the searching and exploration process in order
to do a better job given the amount of information we have.
- So, Liat, you said that you had worked with a little bit
with U.S. PTO helping an appellate project.
How was your experience dealing with a public sector agency
different than your experience dealing with
private sector clients?
- So, in order to do the work obviously with U.S. PTO,
we did it to (mumbles) contractors.
With Deloitte, we partnered with Deloitte.
And the goal was to implement the whole system
using internal system with U.S. PTO.
So, we couldn't work with AWS cloud computing.
So, basically there were a lot of questions about
the networks as you could see.
We had previously discussed about whether we should
open the system right, and allow more transparency.
Obviously, nobody asked us but in terms of for government,
obviously opened code for (mumbles),
but this is one issue that came up.
- So, the PTO got to see the code.
But then the question is should the code be open
more generally I take it, that's one question
that you had to think about it.
And we'll talk more about that I think in the general Q & A
but this exactly what was discussed in the last panel.
When you get a private contractor, one of the questions
that will come up with the agency and suggests
should come up always is will the code be open or not.
Great.
Ian.
- Okay.
Pretty soon we'll have slides (laughs).
There we go.
(laughing)
Okay, Google, where's my slides?
(laughing)
So, I'm Ian.
I run the Google patent search engine team at Google.
I've been there almost five years now.
So, Ed did a fantastic job of the intro to AI, the history.
I'm just gonna run through a little bit more,
focusing on the current advances, AlphaGo, Translate,
things like that.
So, we think about very briefly, what is AI?
Make intelligent decisions by reasoning.
You think human decisions every second of the day.
What am I gonna say next?
Where am I gonna drive tonight?
Something like that.
Humans make decisions everyday by reasoning.
So, here's some decisions I had to take a few days ago,
yesterday when I flew in.
When should I leave for the airport?
Is there gonna be traffic?
Is it a holiday?
Et cetera, et cetera.
There's a weather delay, should I switch my flights?
How much is gonna cost?
How do I balance all these different, the costs
and rewards, all my objectives?
Those are sort of typical tasks that you think
an AI might be able to handle.
And in this case, each of these would be
what's called narrow AI.
You'd have a single system designed to solve
one of these tasks.
So, you might know on Google maps, they have this,
your flight is in an hour or two,
you should leave for the airport now.
They might give you that little recommendation.
So, that's an example of narrow AI.
And the algorithm behind that could be fairly simple.
You could have fairly simple narrow AI
that solve very important problems.
So, here as a human, you're again learning to balance
this cost and reward using your own learned probabilities.
So, you know roughly what traffic's gonna be
based on experience, you sort of know security lines,
things like that.
Now, you replace all those learned probabilities that you
have with what a machine can learn.
So, machine learning is very good at give it some
training examples and come up with exactly
those same probabilities.
Given traffic times overall of history or say
even the last day or two, you can figure out
how long it's gonna take to get to the airport.
So, this is really where machine learning, as Ed said,
machine learning is starting to take off in
the past 10 years.
And we look at where machine learning has fit into
all of the existing frameworks.
So, for example, this is a case from AlphaGo.
AlphaGo mixed things, AI algorithms that have been used
in the past, for a long time, computers, et cetera.
And they replaced a key part of the system with a new
machine algorithm.
So, part of playing Go is there's a huge number of
possibilities for every single move you can make.
And then as you make that move, there's a huge number
of possibilities, what happens after that.
There's a standard algorithm called tree search.
As a computer, you have to tree search through
all these different moves to figure out what's the one
that gives you the highest probability of winning.
Before you'd have all these human created heuristics
for if I move here, for example in chess, if I move here,
if I take this piece, I think I'm gonna win the game
70% of the time.
This is sort of what you had with Go, but it was
very tricky to do with Go.
So, instead, if you replaces these human made heuristics
with machine learning algorithms, that was the real
advance here.
It made the machine much better at evaluating
the current game state of saying, given the whole
board of Go in this state, who do I think is gonna win?
As well as saying given this board of Go,
what are the next probably moves I think I should make?
In that way, the AlphaGo algorithm was much more
effective at traversing down the tree to figure out
which move to actually make.
Here's also a new example.
This is the internals of the new Google Translate engine.
It had been based on statistical machine translation
for years, and more recently they moved it entirely
to using neural networks.
And the real key there, in the middle there,
the real advance was something called attention.
So briefly, how Translate would work is it would read in
a whole sentence like a human would, and then it would
output the words one by one.
And as it's outputting the target sentence,
what you wanna translate into, it's looking back
at the previous sentence.
It's looking at specific words in the previous sentence
and saying this means this, this is the exact
translation for this.
Previously, there was a bottleneck, the machine would try
to memorize the whole sentence at once and then spit it out
without looking at the previous, the input sentence.
That was really the advance there and you can think of that
sort of like AI in that there's this
decision making process.
There's this output the words one by one and search through
this tree of all the possible translations and figure out
what the actual output sentence is that gives you the
best possible result.
But what most of what you see in today's systems
and you know, many of your existing implementations
of AI are these simpler, what I'll call simpler AI.
Here's an example for ranking documents.
You might have these two systems, these two machine
learning models that you've made.
What's the similarity between a query and a document?
And you have another model that says
is this document spam or not?
Those are things that you can gather training data for.
And then you combine them together.
A human can combine those together with their own algorithm.
So, in this case, this is an understandable algorithm.
The final ranking of results is made up of small
understandable parts for the human can go and inspect.
In many cases, the output is simple the output
of this simple AI is simply take the top classification
result that the machine learning model returned.
Or take all the classification scores where it's
greater than .8 or something.
So, there's tasks to solve in patents.
And we wanna see our any of the recent advances,
the AlphaGo and the Translate, are those similar or
different to the things we have to deal with patent search?
So, there's two main objectives of course.
Classification, you take in a new patent document.
Or it doesn't have to be a patent document.
In Google patents, we've classified all of the
non-patent literature.
Google Scholar, Google Books.
You take in some document and you wanna output it
into the patent classification space.
There's 260,000 possible labels.
It's a huge output space.
Now, for this specific one, there hasn't been much
and for search as well.
Search is a more traditional problem but there's again,
that she raised by Liat of the key words being used
are obfuscated.
Generally the concepts that you need to actually search
for a specific technical concept, it's not a simple word.
It's not movie times, it's not SFO airport.
It's the combination of five or 10 different individual
topics that you wanna find in the same document.
So, it's much harder in that case to get good
relevancy results, especially when things can be described.
You can describe a computer, or you can describe a machine
that has a processing unit, et cetera, et cetera.
So, there's issues in large document representation.
A lot of the research has been focused on images
but not necessarily what to do about large documents.
A lot of the text classification tasks, a lot of the
research there has gone into say classify Yelp reviews.
Is this is a positive or negative review?
Did people like this restaurant or not?
Those are sort of short documents that have a small
number of classification codes.
And there's typically a small number of input words
that will make the output classification
positive or negative.
Part of this is simply because there's not a huge corpus of
except for patents, there's not this huge corpus of
finely classified, a huge corpus of training data
that we can actually use to learn.
So, that's where we are sort of in a privileged position
to learn from this for large text documents, and to actually
make some of the research drives in this area.
There's the training data and then the
last piece is interesting, simulation.
When you think about AlphaGo and you think about
self-driving cars, the real advances there is that they
were able to simulate their environment.
They could run unlimited number of games.
They could run matches against themselves.
Or they could run say cars in simulation.
And they could learn over time much more than what
could actually happen in real life.
And so, that's an important thing to maybe keep in mind
if you wanna train algorithms.
Am I am able to simulate to this to again exponentially
more training data?
Here's another I would say two example ways to apply AI.
You have the top one which has humans not in the loop.
And this is really say, the self-driving car case.
Humans or in our case, when we classify all of the
Google Scholar documents.
There's humans not in the loop anywhere.
There's just simply too many documents.
You can't even have a human checking the top result
of each one.
In the bottom one, you have the humans in the loop.
This is where the algorithm will give the human
here's my best possible, here's what I think the top
classifications are but you can make the final decision.
You can see if I made an error or not.
And that also comes into healthcare possibly
and other areas.
So, now I'll quickly go through what we're actually doing.
So, the goal of my team is to make patent information
universally accessible and useful, which is a broad goal.
(laughing)
So, we've been doing this with Google patents
for more than 10 years now.
And over time, it's grown slowly.
We're a small team but very dedicated
entirely to this issue.
We get to focus all of our attention on just this.
So, part of that is our machine classifications.
This is a screen shot from one of our models
that's trained a couple of years actually.
So, this is trying to classify this patent document
with the labels.
Take a full piece of text, figure out what the
actual labels are.
In here, there's an interesting,
all these are very detailed.
There's an interesting, the top result is wrong.
And when you look at the classification scheme itself,
there's a special rule that says if it would have gone
into this category, but it's about in this case, sewage.
It should go into the next category.
So, in this case, the second result is actually
the right one.
So, this is an example of you might have this
underlying machine learning algorithm that doesn't
necessarily understand the business rules on top.
But the underlying algorithm is still very useful.
And if you can apply these business rules on top
of these probabilities, you can get a good result.
We've also recently launched sort of semantic search.
So, this uses the same sort of, it uses an embedding model
trained on this classification.
All of these results are actually released publicly
as part of our data sets.
So, you can go in and you can fetch our raw machine
learning factors.
And we did this in part to move forward the whole
patent search ecosystem.
If we can give away some of this, we can make patent search
more efficient.
We can improve patent quality.
And we as Google, we don't have to be the ones
to make the best patent search system as long
as patent search, patent quality overall gets approved.
So, this is this Google patents public data sets effort.
A free collection of 17 countries worth of patent metadata.
There is maybe 90 million patent publications in there
that you can use to train machine learning models
to analyze for (mumbles), et cetera.
There's a new effort also prior art archive.
This is to collect some data from companies
that was traditionally not available.
Old product manuals, mostly old product manuals actually.
(laughing)
Stuff that you want examiners to search,
that the companies don't necessarily put online.
Okay, so this is important.
If you wanna do a thorough prior art search,
you need machine readable data.
You need to access the full text.
There's examples also internally from Google's
portfolio (mumbles).
We use machine learning internally to create
these landscapes.
When you're at Google scale, you have I don't even know
how many patents we have, but tens of thousands of patents.
And you can't necessarily manually go through and
reclassify all your patents into a new area.
You wanna see what you have in certain areas.
So, you can use machine learning to generate these
landscapes on the fly.
To sort of classify your own portfolio.
And I'll end with it all starts with data collection.
With patents, we got extremely lucky that we just
happened to collect a machine readable format,
the process of the examination.
We were able to collect classification codes, citations,
the full text.
And all of this is extremely useful for training
all these machine learning algorithms.
Now, I want you to think about what happens if we
collect data specifically for machine learning algorithms
that might not have any use as part of the examiner's
day to day life.
It might not actually make a difference in what they do
but by making one little notation in their file,
is this a 102 or 103 rejection in readable format?
It lets us train much better algorithms.
So, I'll stop there.
- So, I'm not gonna ask questions so much as to do a
little bit of clarification for those of you who are not
patent geeks.
So, 102 and 103, both Liat and Ian referred.
Those are rejections an examiner would make because
a piece of invention for which an application
had been submitted was either not novel or was obvious
given the art.
And I think that one thing I should also clarify
is that Liat's company has been able to look at,
so the rejection data that examiners use is
publicly available and you've been able to use that
to train your algorithm.
And I take it that you have done the same, Ian.
- [Ian] So, we've looked at the data and it's very valuable.
- And then the question you're asking--
- [Ian] Part of the inputs too
are next generation algorithms.
- Right, and I take it then you're asking is
that was sort of examiner generated data,
not so to say you're putting into but you're asking
what if generated data that was more specifically
tailored to some of these purposes.
- Yeah, so for example, in classification,
if you had the examiner, right now all we get is this
huge piece of text as classified as this category.
As part of the examiner doing the classification,
if they could say label for us, here's the section
or here's the words that made it go into this class,
or maybe some sort of machine readable description
of here's why it wasn't this class.
Here's why it was this class.
Or if in searching, I looked at all these results,
and right now, we only get the citations, the good matches.
This is the closes prior art.
We don't get from examiners I specifically looked
at this result, 'cause the problem is the citations
are not extensive.
We don't get I looked at this result and it is specifically
not similar to this.
So that's data that could be collected as part of the
examination process.
'Cause that almost fits exactly in the examination process
if the tools were just there to collect it.
You could probably think of other examples that
the examiner, it wouldn't benefit the examiner at all,
but it might help train a machine.
- Uh hmm.
Great, so with those clarifications, Scott, would you
like to react to or comment?
- Yeah, help (mumbles).
(laughing)
I'll like maybe start a quick little intro
and then kind of jump into reacting to some of that.
Scott Beliveau, as for comments of my own,
they're not, don't construe them as policy of U.S. PTO.
Just kind of throwing that out there, I am here for PTO.
So, been at the patent office for 15 years.
I started as examiner, was a manager, done crowd sourcing,
number of executive actions on some of the data collection
and things like that.
Now, I'm in the CTO office sort of doing big data,
open data programs, addressing some of the issues
that you both raised really.
We did, I think it was back in October, November,
where we released that data for the 102, 103s.
Nice to hear people are using it.
That's always great feedback for us to get to be able to
do more of that kind of cool stuff.
In terms of what's our mission, is clearly issuing
timely, quality patents.
And as Arti had kind of intro'd, we had 600 filings,
there's about 1.2 million claims a year, 8,500 examiners.
They're very highly technical examiners.
We have PhDs, they're lawyers, they speak their
own language as lawyers kind of in the patent universe.
So, our challenge really is given that fixed amount of time,
how do you get consistency between examiners?
How do you get them to really be able to search
the more complicated knowledge, ever growing knowledge
in a fixed amount of time?
So, we try and look at using AI and techniques really
to make what, these are some things we're looking at
from a research perspective.
How do you think the adversarial process
of patent examinations?
The attorney is trying to get as much as possible.
The examiner's trying to shrink the scope down.
How do you make that process more predictable?
AI is a great way to make that predictable.
So, if you file something, you should have an idea
what you're gonna get?
How do we do that?
One way to do it is through enhanced transparency
which I think is something you know, you guys
were talking about earlier.
Are there ways that we can make that data more transparent?
The other is making the process more consistent.
We have two examiners.
If you give them both the same case, you should reasonably
expect the same answer.
That's a fair thing.
Are there ways that we can use or leverage AI to do that?
Their is really, and getting to some of the other issues
with if we had this enhanced instrumentation data,
how people are making those decisions.
How we build a learning organization?
How can we leverage AI in your example of the
car-driving of where you're gonna go?
How can we do that to help not only in the pre-grant
process but the post-grant?
How can we learn from our post-grant activities
after patents go out and they come (mumbles)?
So, for classification,
why is classification important to PTO?
Well, it's an administrative data function.
It's route cases, if you get the right cases
to the right examiners who have that expertise.
It's how we assign time to all of them.
And it's something that we spend a lot of time,
effort, money doing it.
So, the more that we can use AI to promote a consistent
voice of classification between all the international
offices, that really helps us from a perspective of
covering that broader range of knowledge or information.
As well as that question of the why.
Why did you put this where you put it?
That's a very valuable point for examiners for searches
because when our examiners are really doing search type
activities, examination really is,
there's two parts to the search.
And there's examination and examination is really
(mumbles) of explaining the judgment or the reason as to why
you came to that conclusion.
So, are there ways that we can be more transparent?
Like track the way, not in a sinister way,
but track how people are arriving that conclusion.
Use AI to promote that consistency of that so that we're
not really just simply giving examiners a fish.
Here's a list of results.
But really helping them assist them to where
to find the best fish.
And then taking that information and providing it publicly.
As our applicants file, higher quality applications,
it improves the examination processes.
- So, I'm gonna ask you a tough question.
So, it is an adversarial process as you pointed out.
The patent applicants want the world and you can't
give them the world and you should give them the world.
So, the more transparent, this is one area
where transparency may not serve public functions.
If you make all your
algorithm transparent, would there be ways for them
to game the system such that you wouldn't find
certain prior art?
- Potentially.
Yeah, I think in a larger AI discussion construct,
there's always that pros and cons of too much or too little.
But I think from a public agency perspective,
we want to enable and empower people to have the best
technology to file the highest quality applications
that they can.
So, if they're using that and the other piece
of the transparency argument goes it's really important
that the construct of algorithmic bias or AI bias,
it's important for us to be transparent as to when
we're using AI techniques, that people know how
we arrived at that conclusion.
What's in there?
'Cause a lot of the AI techniques are cool stuff that
comes out is made by hi-tech companies.
And those hi-tech companies are applying for patents
and we wanna make sure that it's transparent,
that we're not introducing and particular bias of
well you used company X's algorithm.
Therefore they get a better shake out of the process
than another.
So, transparency is very important to our process.
- So, that's interesting.
So, that's another flavor of nondiscrimination,
saying nondiscrimination with respect to individuals
in particular, traits.
So, you're talking about essentially either,
you wanna fend off the appearance of corruption
or some sort of allegations that there may be
bias in that sense.
Very interesting.
So, I have many more questions to ask about transparency
and we'll get to those.
But Alex, did you wanna comment a little bit
from your perspective regarding how you use data
and machine learning at the Bureau of Labor Statistics
and whether there's anything?
Are there any cross agency learning opportunities here?
- Definitely.
Thank you for inviting me to be here.
It's great to be part of this discussion.
We don't, to my knowledge, do a lot of intellectual
property search at the Bureau of Labor Statistics.
Although, that may change as Scott was mentioning,
there have been some talks about some projects.
But what we definitely do plenty of is classification
and text classification.
Any time you see a statistic referencing an occupation,
an industry, a product category or about a million
other things, there's a classification happening
behind the scenes to make that possible.
And often it's a classification based on text.
The reason for that is that that's the most natural way
for people to express these things.
When we send a survey out to someone asking them
what's your occupation, they don't say "I'm a standard
"occupation code 372011."
They say, "I'm an environmental services technician,
"or I'm an associate, or I'm a laborer."
And we have to figure out where to put that.
It's a very natural way for people to exchange information.
It's just not a natural way to compute the number
of janitors in the economy.
So, classification and text classification
is a very important problem for economic statistics
because so much of what we're interested in
is most naturally expressed in that way.
It's also an important problem because until recently,
the best way to do that classification in most instances
is manually.
We have people now, we have a small army of people out there
who spend their days reading these job titles,
descriptions of business activities, and assigning these
classifications by hand.
When you are collecting millions of data points each year,
that turns into a big resource cost.
Also, (mumbles) challenges of sort of consistency.
You show the same job title to five different people
and sometimes you get five different answers.
So, there's been a lot of interest in sort of improving
and sort of using new techniques to address that.
What's happened in the meantime over that last 15 years
or so is that we now have more previously classified data
in digital format.
And we now have in many cases lots of data and previously
classified data in digital format.
And when you combine that with modern machine learning
techniques and powerful computers for processing
that information, you now often have a very relatively
simple and effective way of automated some
of that classification.
And that's what we started to at the
Bureau of Labor Statistics.
The best example of that is with our survey
of occupational injuries and illnesses.
Which among other things collects about 300,000 written
descriptions of work related injury and illness each year.
These are descriptions which are collected
from essentially administrative data, OSHA logs.
As with much of the text information we collect,
we have people go through and assign classifications
to indicate the occupation of the worker, the cause
of their injury and so on.
What we found with machine learning is not only
can we with relatively little work, build systems
to automate that, but these systems in fact work very well
and in fact, they assign these classifications more
accurately than our trained human staff.
So, as a result, over the past few years, we've gone
from manually coding all of this information to now
automatically coding the majority of that information.
That means we can process more data faster
at higher quality.
It means for our staff, that they can spend more of their
time on other important tasks and frankly often
are more interesting tasks.
But I think most importantly, it also means we now
have a huge opportunity to apply this technique
in a whole bunch of other places because there are a
whole lot of statistical programs that spend a lot of
time on this task.
And so, that's something that's happening
with the Bureau of Labor Statistics.
There are a whole bunch of projects that have just started
in the last few years.
And I think it'll be very interesting to see
how that progresses.
- Great, so I have only three questions before I open it up
to everyone.
So, one question is so we did spend the last couple of,
speakers (mumbles) about classification, excuse me.
So, are these dynamic classification algorithms?
So, one of the concerns that we've had with the
PTO classification, I can speak to that specifically,
is that often times, the new technology classes
are introduced well after the time they should
have been introduced.
So, can machine learning speed up the process of
classification so that we have
classification that's more meaningful?
- [Alex] I mean that's something we're interested in.
And what we do in sort of the economic statistics world
is we periodically update these classification systems.
There's always new occupations being created.
There's new industries emerging.
And so we try to keep up to date with that
and when we go through that process of updating the
classification system, it's a very heavy research process.
We've started looking at some techniques automatically
identifying new sorts of clusters if you will
of things that are emerging.
But that's certainly something of great interest to us.
- I'd say absolutely.
So, one of the pieces right now, the US Paten Office
had transitioned from USPC which was primarily developed,
maintained by the US Patent Office to a cooperative
classification system with our European partners.
So, to that extent, there has to be some, we have to work
with our partners and negotiate ways of doing changes
to that classification.
And that's sort of where some of the work, research
that John over there is doing a lot of work on
if he was to wave.
Ways of saying can we leverage AI technologies as an
entry point into let's having, 'cause as we've looked at
classification between countries, between different tools,
outside of the group level, it gets very like,
it diverges a lot, particularly on the subgroup levels.
Can we use AI, can we get to a point of developing
algorithms to create a starting frame of reference
as to how you do the classification.
Then from there, migrate to I think what you're alluding to,
is okay a new topic is coming out.
Because we've sort of agreed upon this framework,
now let's be a little more (mumbles) into it.
- And one of the things that leads to me to focus
on that issue is that there's a lot of empirical data
suggesting that the rate of recombination of classes
is where we're getting lots of new invention.
And there's really good economic literature on that point.
And so the fact that recombination is happening so much more
strikes me as the reason to have dynamic classification.
I don't know if Liat and Ian, you had thoughts on
this question.
- I think it's important, this is an important question
generally across AI.
This is a case where the input distribution is shifting.
You've trained your classifier on the patent corpus
as it was today, and in a year from now, there's new
language that pops up.
New areas are getting combine that weren't before
and the underlying statistics of your model are gonna say
healthcare and AI would never combine if were
20 or 30 years ago.
Maybe more.
But today, you'd see a huge number of classifications.
So, the bias is built into the system because the models
are trying to optimize in general classified based on
what I've seen with the statistics that I've seen.
Those need to change over time.
So, you either need to retrain the model or it's the case
for keeping the human somewhere in the loop.
The model can help you narrow standard possible
classifications to a couple, but if you keep the human
in the loop, then they can always adapt to future
changes in the distribution.
And also, it keeps it consistent as well which is important.
- And the other challenge sort of inside baseball
in the building is if we had a model develop
a classification system that was the classification of one,
every document was essentially became its own subclass
to that level, that creates a huge administrative challenge
within an agency to say okay, did you search everywhere?
Well I searched in one.
(laughing)
So, I'm sure there's a sweet spot to that kind of aspect
of dynamic classification as well as kind of going to
Ian's early point of we're looking at ways to be able to say
not just a document was here, but why, what exactly made
the document here?
And keeping the human in the loop to say okay,
you as the classifier, you had this choice of where
to send it, what particularly did you look at
within that document to send it to a certain bucket
as opposed to another bucket?
And then using that type of information to build
I think some of the stuff Ian was looking at of that model.
It's not just simply a positive, but what did you not look,
what did you look at but not decide to go in that,
I think the judgment in that direction?
- And we at AI Patents, what we've done basically is
we realize it, AI right, you learn from the past
what predicts the future, right.
And this is what lead to us to the idea that what's
important is how you hold a document.
And this helped us both in classification
and as a search process, because at the end of day,
you cannot break this huge document to keywords.
And the ability to look at the whole document,
we're actually able to classify better even in today's
patent (mumbles) since they do the same technology
even though they're classified into different IPCs.
So, I think the human into it again, I don't believe
that in the future, computers can do everything in terms
of classifying or searching.
'Cause the idea is to present the user the reason
why you got these results or why these two patents
actually found to be matched in the same classification
in order for the human to make the decision (mumbles).
- I just have one another question and is just returning
to a theme from the beginning which is
you're a for profit company and you contracted with the PTO.
And I take it there was some questions about whether
the source code should be made available or not.
So, is there a way to have some level of transparency
for the public at large by (mumbles),
so having the government specific source code
and other code for your private sector clients?
Scott and Liat, perhaps you can speak (mumbles).
- Yeah obviously serving the government and then
understanding why the transparency is so important
and the ability to provide the agency to build on top
of our code is something that definitely should be
a good approach.
Because what we kind of provide is basically
also the ability to search through long text documents.
As well as the ability to use the cumulative knowledge
of these experts.
These two things are very important to (mumbles)
of the government, the need to use it in the way
that they want is important.
For that reason, obviously opening the code is important.
And we can find ways in which as you were saying (mumbles)
but for the government use that is made with that,
you can have the code.
- [Arti] Scott.
- I kind of feel that one of the kind of open,
I think we touched on a little bit before is it's important
as an agency that when we're using the semantic
or AI search capabilities that the public is aware
that there's, everyone has that level playing field.
I've seen some models where which was mentioned earlier,
some companies provide this much in the open
and then there's this is our unique special sauce
that we keep proprietary, I've seen some different
models in that.
Or from a government agency, we wanna make sure
everybody is aware of kind of what's going on.
And the open source aspect of it, another challenge,
I think you touched on a little bit in the earlier part
is as an agency, as IT expenditure work in three year
budget cycles, so we have to plan our budget out for three
years, and then when we try to do procurement activity,
it takes us usually about eight months to do a procurement.
So, we're always trying to think like three years in
advance what are we gonna need, how much we're gonna spend?
And the other challenge that we have is our limitation
with respect to cloud services.
We really as an agency right now, we're still at a point
where it's gotta be in the data center.
So, therefore when we look at this open technology
versus closed technology, because we're always
looking at three years, a three year window, we tend to lean
toward the wanting to use more open source technology.
- Great, so time to open it up to questions.
And we have a fair number.
So, let's start with Wes, because--
- [Wes] So, in your evaluation, that the way you talked
about the evaluation patents take advantage
of examiner decisions, do you recognize the possibility
of systematic error on the part of examiners?
And then maybe going a step further and looking at
downstream of litigation and validity judgements and so on.
But the trouble and difference between what you're doing
and say AI applications in autonomous vehicles,
or even medical applications, there's no objective reference
for saying this is right or wrong.
And you have people getting run over by cars.
That's an objective reference.
But the decisions around novelty not obviousness
and so on are invariably subjective.
And this underpins a more general question.
Is there a way to get AI evaluation of patents
to the point where it actually does a lot better
than examiner, systematically better than examiners?
So, improving on examiner or judgements.
- Actually, that's a very good point here in terms
of our approach.
So, the AI if you remember in my diagram starts with
how to represent a document.
This one leads actually to how the search is being done.
Before taking into account the (mumbles).
But what we would like to see is to, in the past, examiners
or inventors missed critical information which lead
to litigation.
We (mumbles) our system to be biased on that.
So, we kind of combined two approaches.
The first one is a viewer search.
Is taking a small document and take it into account.
Don't take out keywords, don't claim it.
Just take it as is and allow the computer to really
scan through all of this document and provide
a purer result.
On top of that, we build AI and the user is
very active in this stage.
He can select which synonyms or which not only synonyms,
like which technical terms he chooses to select
from what the system suggested.
As well as he could select how strong
would you like to take the AI part in the results set.
So, the combination of a strong search engine
with AI can allow the user to define how strong
it will take into account, this is the important part.
Obviously, we took the examiners because these are based
on a long process of rejections, right.
It's more, better adapted, more valued data than in the
search, okay this one is relevant, this one is not relevant
in end user, but it can still miss a lot of
valuable information.
- Yeah, so I think it's a really interesting point
that if you think of say we're training on the citation
data that examiners use.
That citation data was gathered through examiners
doing Boolean searches looking for exact keywords.
So does that mean that any model that we train
is gonna look for examiner keywords?
And it's a really interesting point.
You'd hope that the examiner would go beyond that
into 10 or 20 searches and come up with all the synonyms
and even then phone a friend.
(laughing)
'Cause they do that, they call up their friend
or they have references that they keep in their little
drawer that they just know by heart, this is prior art
for this and maybe none of the keywords match,
but they still cite that.
So, there's sort of an out and we're hoping that examiners,
and given all the examiners, and given EP and all
the other countries together, if you funnel that
through such a bottleneck when you're training
your algorithm, you get rid of all of these individual--
- [Wes] But that's the issue with the distinction
between systematic error versus random error.
If the errors are indeed systematic of a particular
character, then that might help with.
- Yeah, so if all examiners are at most, 30%, I don't know,
some number always made these errors, or there's some
searchers were always select one keyword for ever--
- [Wes] Not always, just--
- Just some yeah, select one keyword.
- It can be in a particular pocket or area within,
it's technology across the board, yeah.
I think it's been said before, in some way we have
8,500 patent offices.
How do you get all of them working lockstep?
Whether it be in looking at the construct of if it is
a systematic error, can we use AI and our other
techniques to identify to that systematic error
such that we then as an agency go back and do the
training or other type of action really at the lowest level
and as soon as possible before it becomes disruptive
into the IP community?
- [Wes] So, can you?
- We're trying.
Yeah, I think that some situations are easier than others
to address that.
And sometimes, a lot of times with examination,
a patent is issued at a certain point of time,
the legal law changes, it's always a moving target.
So, you have to say, well was it systematic error
looking back?
Well, within hindsight, yeah it was, 'cause we followed
a particular guide.
101's a good example.
There was a particular guidance for 101.
The agency followed it and the rules changed
because of some other, and then we go back and we try to
look and fix that.
In terms of your other question about could the AI
be better than the examiner?
We're looking at it more as an agency, is the machine
in partnership with the human in having superior results?
And I think there's a lot of research that's kind of
shown when you do just machine versus machine
or machine versus person, versus kind of teaming up.
Where we're looking at using AI to sort of surface
or bubble up or suggest things to, maybe on the
car drive example.
Maybe you should stop at this burger joint because everybody
who's your friend stops there and they love it.
Maybe you should check it out.
Providing the examiner so that they're still in the
driver seat making that judgment call, because it is
a very as you said, there are certain very
judgemental processes that go on in it.
But can we promote the consistency of that judgment
so that people cover most information, most facts at hand
at that time to make a consistent judgment?
- I think, the issue of systematic bias,
there are sort of second and third order effects here.
If you put such a system into effect and you say,
here examiner, the results you see are just what the
algorithm modeled up.
And then you start training on that data again
in the future, you never really have the examiner
doing a broad search on their own.
There's no possibility of something else
getting into the system.
So, it's important, same for classification.
If you're always showing the examiners,
say at the top results, you might get into this
dangerous feedback loop.
And the stats and the cost savings and the accuracy
might look fantastic for the first couple of years,
but over time you'll lose the importance of human
classification aspect that's needed to retrain your system.
- And that's where you talked a little bit about
giving a fish versus training them, doing ways to make,
find the better fish.
Ways to become a better fisher person.
- So, I'll just make one editorial comment.
One systematic bias that we know about is
failure to (mumbles) non-patent literature.
So, you know that, but we all know that.
Alright, Andrew (laughs).
- [Andrew] It's Andrew Chan, UNC Law School.
I've heard you talk a lot about the role of classification
system in the examination process.
But I'm gonna disguise advocacy in the form of question.
(laughing)
I'd like to ask to what extent you've paid attention
to the public notice function of the classification
that's attached to the patents that are issued?
Is that the public when they are deciding
they have freedom to operate, they're viable
for (mumbles) of any patent that's issued,
but they may only have the resources to focus on
the classifications that relates to products
that they are selling or planning to sell.
And you know, the reason I think it's important
to attend to that simultaneously through the
examination processes, from a study that I did
and actually relating to what Ian was talking about,
that prior art can come to the examiner's attention
either through literal keyboard searching or
they may have something in the drawer.
I studied that and found that, it was an invent study
looking at the rollout of east and west
and the computers going on in the examiner's desks
around the turn of the century.
And what happened was that the citations that were found
through keyboard searching tended to be more often
quote classified than those that they sort of
knew of intuitively.
That suggests that as you're, that there's kind of a
policy lever, as automation is pushing out the trade-off
curve between recall and precision, information retrieval,
there's kind of a slippage for loss of precision
as you take humans out of the loop.
And if you're thinking about how much in terms
of human resources but where you want to be on that curve
to improve internal processes of examination
within the patent office, you may have a side effect
in terms of utility and public notice function
of classification.
- So, classifications, they're very useful internally
for the PTO to actually route out applications
and to run their whole function.
But they're extremely useful especially on the
international scale to actually do the search itself.
If you look at examiner search logs,
if you look at the public pair data,
you can see what examiners are searching.
You can see they usually always do class code
and some keywords.
They always restrict into some class code.
So, there's a real danger there that if the class code
is not extensive enough or what you're studying I guess,
they're finding outside these class codes,
that's a real danger to what they have.
So some of the systems that we've designed,
they take into account the class code, and if you've seen
the scholar classifications that we do,
we tend to optimize more for recall.
You'll see a lot of classifications that could apply
to many different areas.
If they were classified like patents, they'd probably
only have a couple.
So, that helps in retrieval but of course,
if you go too far, you make the classification worthless
for examiners, because they use classification almost
because their relevancy ranking for all tools
are not good enough.
They have to arbitrarily restrict to this classification
code and some keywords because the relevance overall
of matching those keywords is not just good enough
or not even there at all by date.
- Yeah, some of the difficulties,
the language of innovation.
I mean, people make new things, they make new words.
They have to call it something because there wasn't a
word to describe what this invention is.
And that's, so examiners are required to do a classification
based search and that's why you see in the search logs,
they start with a classification and that's because
just purely doing keyword search if somebody can't,
I think somebody gave an example of like LED
was a word from the 1800s.
Really didn't have anything to do with LEDs now.
It was something like that.
And that's why they do that classification
more faceted type search and build upon it.
Now, the risk being is you get a DNPL issue
where in the DNPL coverage--
- [Arti] I'm sorry, we tend to use acronyms.
(laughing)
- That information isn't necessarily within the boundaries
of the PTO east-east system that you mentioned.
Isn't classified or corpused in that manner,
so examiners then have to sort of do this
and they go a little more keyword.
Now, one of the other challenges that you'll see is
within that, if you have to then do the swivel chair
and type in the new thing, it's not necessarily
you're off the main examination tool system.
So, your recordation of the fact that you did this
isn't always reflected in the record of the case.
And that's where you can get some sort of
a perception, well that people are not searching.
They may have, they may not have.
Then if and then didn't find anything or they just didn't.
- [Arti] Let's see, I think Steve had his hand up
before Jerry, but Steve.
- [Steve] I'm wondering if it is the case or conceivable
that your applications could be used to improve decisions
remote from what you do in your jobs?
For example, could it help
anticipate what kinds of people the PTO has to hire
in the future?
Or it could be used to inform OSHA on what it should
be looking for to regulate in the future?
- That is something we are looking at.
And it kind of goes to the public notice of classification.
It takes on average about three to five years
for an examiner to get fully up to speed
and kind of be completely self sufficient
as a primary examiner.
So, if you think about a three to five year
training hiring cycle, we need to make sure that we have,
where the work is going in three to five years
so that they develop an expertise in that technology
that's gonna show in three to five years.
So, we are, it is something that we do look at internally.
And how can we create more?
And this gets to technology landscaping and things like that
of where should we be allocating our resources
three to five years out?
How should we be training people?
What technology areas to be focusing on?
So that when that technology kind of bubbles up,
we have adequately trained staff who have enough
expertise doing those complicated judgment calls.
- I would add one sort of interesting thing
about the technology, at least that we're using
for classification.
Is that it's a very general technology.
I mean, you're sort of feeding in previously classified
information and it's generating predictions.
You can apply that to it turns out all sorts of things.
You can apply to figuring out whether the car's
still on the road or not.
You can apply that to translating from one language
to another trying to figure out what the next word is.
You can apply that to trying to automatically
detect errors in the survey data that we're collecting,
or trying to predict where the next injury
is going to happen.
So, there's I think one of the most exciting things
about sort of machine learning is just how widely applicable
it is to the various tasks that are out there.
One of the challenges in sort of sharing the machine
learning systems from at least our perspective,
is that the machine learning system is a combination
of both the methodology that was used to build it,
but also the data that it was trained on.
And in some instances, you can reverse engineer
some of that data.
So, when you collect data under a certain
pledge of confidentiality, that becomes problematic for us.
There's research on ways to obfuscate that but that remains
sort of a big problem in sharing models trained on
protected information.
- [Arti] So Alex, you have a different type of data set
than these folks have to deal with.
You have lots of privacy issues presumably.
- Yeah, most of the data that we collect is collected
under a pledge of confidentiality unlike sounds like
the patent information.
So, we have to protect the identities of the people
that provide us that data.
And so.
- [Arti] Jerry.
- [Jerry] Wes anticipated my question.
So I'm going to push a little bit harder about the...
I'm very excited about the possibilities of AI
with regard to novelty and speeding up the process.
I think that's fabulous.
I'm very skeptical about what AI can do with regard
to non-obviousness because I'm very skeptical
of the standards that the patent office is using.
(laughing)
There are about 600,000 applications and last time
I heard award about 65,000 patents a year.
So, you can say, well 10%, that's pretty conservative.
I think it's astronomically un-conservative.
65,000 non-obvious patents.
I find that absolutely unbelievable.
And the fact that we have all this money being wasted on
costly lawsuits that are everyday invalidating patents
from right, less, maybe a bit less with post-grant
opposition procedure I think is wonderful
and long overdue.
But I'm curious to know if you think that
a skeptic like me can be persuaded that I can improve the
quality, the non-obviousness standard in the patent office?
- I'm trying to think of (mumbles).
To a degree, I'll say.
Ultimately, we are trying to get the examiner
to be able to look through the most amount of art
to make that judgment call.
Is the machine saying, combine A and D?
No.
That's really not the things we're looking at.
We're trying to provide the tools if an examiner
and their judgment says reference A has most of it.
I'm gonna now look for the missing part,
and if I find the missing part, that's the examiner's
judgment call.
And that maybe goes a little more to maybe some of the
training or policy or some legal aspects.
But we're not necessarily looking at an AI
decision making process to say yes this is obvious.
And even with novelty--
- [Jerry] Excuse if I just (mumbles) for a moment,
but it seems to me is that what the examiner is lacking
is any faith in competition.
I mean, used to be before--
- [Arti] I don't think that's related to the
AI piece of things, yeah, so but I think that
Scott has conceded that.
So we have time for one last question.
And the fellow in the back.
- [Man] This is a different question.
And I apologize but it will help me understand
the capabilities of these systems.
Do your systems either learn or impose a metric
on the space of patents?
And similarly do learn or impose a metric
on the space of job categories?
- Job categories.
- Well, for job categories, we do have a limited
number of categories.
And these are defined by our standard sort of
classification system.
So, it is constrained space.
- [Man] Okay, (mumbles)?
- Yeah, I think it's like all things in patents,
it's complicated (laughs).
It depends on what is is?
I think there are metrics that we look at
from a consistency perspective.
It could be particularly rejection types.
Is one examiner doing this or that?
A lot of it is sort of more exploratory.
- [Man] I apologize, I was unclear.
A mathematical metric is a way to kind of distance
between two patents.
- Yeah.
- [Man] (Mumbles) patents put together and these
are way far off.
- Yeah, so my esteemed colleague over there, John,
can probably tell you a little more about the ins and
out of the output, the specific of the algorithmic
nature of it, 'cause we are using a number of different ways
of protected search like that.
- So for us specifically, I did a talk last year
that's online.
The EPO search matters.
So, the keynote goes into some more detail.
But yes, we do embed all of the patent documents
into this embedding space with the vectors
and you can compare the vectors to get a distance
to see how similar how they are.
And then we also embed the classification codes
under the same space.
So you can compare class to patent, class to class,
patent to patent, et cetera.
- We are basically representing every document
in this vacuum.
But we not saving the relationship.
Everything is running on the fly.
And it is based, once you generate the scoring,
you can put the AI on learning for (mumbles).
- [Man] Thank you very much.
- (Mumbles), did you have a question?
No.
Okay, so we have run out of time.
And I'd like to thank our panelists.
(audience applauding)
We have lunch that you can pick up in the servery
and we will have about 15 minutes to do that.
And then at noon, we will once again have work to do.
(laughing)
For more infomation >> Hombre se hacía pasar por agente de migración | Al Rojo Vivo | Telemundo - Duration: 2:19. 
For more infomation >> Dramáticas imágenes de accidente aéreo en Honduras | Al Rojo Vivo | Telemundo - Duration: 2:03. 
For more infomation >> Detienen a sospechosos de muerte de mujer policía | Al Rojo Vivo | Telemundo - Duration: 0:42. 
For more infomation >> Acusan de negligencia médica al 911 | Al Rojo Vivo | Telemundo - Duration: 1:13. 
For more infomation >> Peligrosa hazaña en moto casi cobra una vida | Al Rojo Vivo | Telemundo - Duration: 0:30. 
For more infomation >> Tensión durante inspección de escuadrón antibombas | Al Rojo Vivo | Telemundo - Duration: 0:37. 
For more infomation >> Meghan deslumbró en su primera aparición como Duquesa | Al Rojo Vivo | Telemundo - Duration: 1:18. 
For more infomation >> Caos y descontrol por mono que escapó en el aeropuerto | Al Rojo Vivo | Telemundo - Duration: 2:20. 
For more infomation >> Muere otra sobreviviente de accidente aéreo en Cuba | Al Rojo Vivo | Telemundo - Duration: 0:53. 
For more infomation >> Secuestró y escondió a su pareja en cajuela de su auto | Al Rojo Vivo | Telemundo - Duration: 0:38. 
For more infomation >> Atentado contra exfiscal sembró el pánico en Guadalajara | Al Rojo Vivo | Telemundo - Duration: 0:46. 
Không có nhận xét nào:
Đăng nhận xét