Handwriting recognition software for indexing?

spencerschumann
New Member
Posts: 4
Joined: Mon Jul 31, 2017 4:40 am

Re: Handwriting recognition software for indexing?

#11

Post by spencerschumann »

Thanks for the link to the article about the BYU research team! I'm trying to contact them to learn more.

I took a course on machine learning a little over 15 years ago while working on my computer science degree. But I must admit, I've done very little with machine learning since then, so my knowledge of it is rusty and I haven't kept up much with the latest advancements apart from seeing articles here and there about advancements in the field.

There have been some huge advances with neural networks, and some of the results are very promising. But at the same time, neural network techniques have not yet fully solved the problem, and I wonder if they may be approaching a local maximum in the solution space - a solution that's good, but not ideal:
handwriting_recognition_solutions.png
(39.37 KiB) Not downloaded yet
Because of this, I'm interested in exploring less conventional solutions. Also, whether I personally work on neural network solutions or not, they will continue to receive a great deal of attention due to their popularity and effectiveness.
russellault
New Member
Posts: 2
Joined: Fri May 05, 2017 5:46 am

Re: Handwriting recognition software for indexing?

#12

Post by russellault »

Like the other posters here, in the past 9 months I have taken great interest in automatic handwriting recognition for the purpose of indexing genealogical records. I took Andrew Ng's and Jeremy Howard's courses and am now working with a deep learning researcher at the Ohio State University on the problem. We have spent the last several months replicating BYU's approach and making it open source. We've been in contact a little with them about this. We just submitted a result to the 2018 ICFHR competition on making a general recognition solution that doesn't require additional training data. We're now trying to research and implement better approaches. All our code is on github, in a repository called Nephi. We welcome any contributors, all you need is access to an NVIDIA GPU.

The best open source platform for handwriting recognition is probably Transkribus, which is funded by the European Union. I am using their platform in conjunction with our code to transcribe hundreds of pages of my own handwriting. While their platform is easy to use and fully featured, they haven't yet made their algorithms open source, but instead offer their platform as a free service, while the project has funding.

If anyone is interested in talking more about these things, I can give you my email.
spencerschumann
New Member
Posts: 4
Joined: Mon Jul 31, 2017 4:40 am

Re: Handwriting recognition software for indexing?

#13

Post by spencerschumann »

Thanks for pointing this out! I found and starred that repo. My own individual efforts, in my free time, have been slow going and not too successful. I'm interested in looking more closely at this project and the techniques it uses.
russellault
New Member
Posts: 2
Joined: Fri May 05, 2017 5:46 am

Re: Handwriting recognition software for indexing?

#14

Post by russellault »

Yes, you are right that there are two problems, segmentation (where are the words on the page/also what is their semantic meaning like birth place/first name/etc.) and recognition (what are the letters/words, given a segmentation). In addition to improving each of these processes somewhat independently, people are also working on methods that do both at the same time. This is definitely possible, and I don't think it's too far off when we'll have methods that can take the spreadsheet from FamilySearch Indexing and together with an image of a document train the AI system. Tranksribus is one of the groups currently working on this problem. Seth Stewart from the BYU team just recently published a paper where with just a few pages of training data a neural network could output where all the handwritten text, machine text and dotted and solid lines are. The paper is titled:

"Document Image Page Segmentation and Character Recognition as Semantic Segmentation"

My collaborator and I are working on one mechanism that allows segmentation and recognition at the same time. It is called "attention" and has been used by a few people in the past 2 years for segmentation and recognition at the same time. We are currently working it into our github repo.

As for using human input to train and retrain AI, I think that's definitely possible for handwriting recognition. The systems can be set up to put out a confidence in the prediction. I think for the time being having the machine be one indexer in the current 2 indexers + arbitration is the best way to maintain accuracy. I think another next step could be to train a machine to be the arbitrator. I think that's definitely possible, but we may still need at least one human in the loop for now.
psventur
New Member
Posts: 1
Joined: Fri Jun 01, 2018 8:06 am

Re: Handwriting recognition software for indexing?

#15

Post by psventur »

Hi, this is my first post on this blog, and I would like to contribute with some ideas that may help doing handwriting recognition (HWR).
In fact I was just reflecting on the way a "human scanner" browse through records (eg vital records like birth, deaths, etc) to identify meaningful info.

While going through a batch of vital records, looking for specific information, our mind first builds a "template" of where the relevant fields are (name, surname, parents, birth date, etc), then "instructing our eyes" to jump to the key record(s) we are interested in (usually a name), thus dramatically speeding up the manual scanning process.
This also works well because most of a typical vital record is made up by "formulas" that are pretty common to the same batch, like "on the day XY, in front of me XYZ , in my quality of civil officer of the municipality of....." - a lot of junk you typically just skip!

I believe that the same thing could be done with an AI engine , by implementing a simple strategy like:
1) setting up a "semantic template" for each specific batch of documents, identifying repeating "formulas"
2) track the output of the HWR for the first few records, fine tuning the template
3) track a few more recognized records , "teaching" the system to improve interpretation.
4) get an alert and intervene if a specific record doesn't seem to follow the template

This may not be the "killer app" that reads all records whatever, but would certainly speed up by at least two orders of magnitude the digitization of handwritten records.

I think it mimics what Mail services do to automatically sort handwritten envelops by destination.
It's way easier to interpret the meaning of a field if you already know what type of info is in there, and a template would just do that.

Thoughts?
Post Reply

Return to “Software Development”