8 Ways to Overcome OCR Errors when Searching Newspapers

Picture

Everyone who has searched newspapers online will fail to find something. It happens incredibly often. The stakes are high for genealogy researchers, where finding an article about an ancestor can make a huge difference in filling out a family tree.

I have often heard researchers say “I can’t find a single article about my ancestor, even though I have searched for hours!”

Laying competency aside as a factor, the biggest reason is that scanning of one and two hundred year old newspapers, either from paper or from microfilm, produces way less than optimal results.

More importantly, one must know that searching through an index created by humans who have read the source material and then typed the index is far superior to having a machine/software scan and process a dusty old newspaper. Yet the massive size of newspaper collections prevents the creation of the index manually. You must expect inferior results and set your expectations accordingly.

Please take a look at the following list, and hopefully some of these errors and anomalies will provide you with some hints to overcome them and actually find what you are looking for. There are many others – but these are ones that I have personally experienced:

  1. Hyphenated words were often used because of fixed width type as well as the experience and capability of the typesetter. Hyphens are less utilized today but were a staple years ago. Take that into consideration if you are searching for a surname or other search criteria with many letters in one word. Try splitting the search into two words where the hyphen may have been normally used.
  2. If there is an “h” in your search term, try exchanging a “b”, since b’s and h’s are quite similar and can “confuse” the OCR process. As an example, searching the California Digital Newspaper Collection for one of my surnames – “Braunhart” yields 1,507 results. Replacing the “h” with a “b”, hence searching for “Braunbart” yields 96 results – for the SAME person. That is approximately another 6%!
  3. For a similar reason as “h” and “b” are confused – the same holds true of “c” and “e”.  I have not had as many difficulties with this pair as with “h” and “b”.
  4. Likewise, lower case m’s and n’s are often confused. The m’s are often converted to several combinations of letters.  Also r’s and n’s can be confused.
  5. I’s in lower as well as upper case can often be converted to slashes or exclamation points and the numeral 1. And vice versa.
  6. if the original newspaper is “dirty,” by that I mean there is excessive ink or the scan is dark – many times spaces will be scanned but not presented as spaces. There are a variety of strange characters that may be picked up.
  7. If the newspaper was scanned and then processed directly with OCR, that is one pass. If the newspaper was scanned to microfilm and scanned again and then OCR’d that is two passes. Thus a two pass operation has the potential to have a decreased quality of results. There isn’t much that you can do about it – but it is nice to know.
  8. This one is not really about scanning, but is more of a cultural challenge. Until the last few decades, women were not referred to by their maiden or given name in newspaper publications. So in my mother’s case, after she was married, any newspaper articles cited her as Mrs. Robert J Marks, or Mrs R. J. Marks, not Muriel Marks. So looking for “Muriel Marks” with the exception of her obituary – would have led to zero results.

So don’t be discouraged by “lack of results” from doing online newspaper searches. You just need to “outsmart” OCR and try various combinations to get to those elusive ancestors. Be persistent.

An additional help would be more crowdsourcing to correct OCR errors and improve the text. An example is reCaptcha processing that is used for Google Books.

Another crowdsourcing example that I personally have used is that of correction on the actual online newspaper site, such as the aforementioned California Digital Newspaper Collection. In this example, registered users can provide edited text that is then incorporated into future searches. Kind of like a newspaper-related “pay it forward.” This capability is provided on that site and many others from the fine folks at Elephind.com, who created the software used by the California collection as well as many other online newspaper sites.

For many more details about scanning, OCR and related subjects please read Scanning FAQ by Project Gutenberg.

Another excellent article is Analysing and Improving OCR Accuracy in Large Scale Historic Newspaper Digitisation Programs, from the March/April 2009 publication of D-Lib magazine.

Good luck – be persistent and have reasonable expectations.



Join the Conversation

13 Comments

  1. Great points all. Searching by Mrs [all husband variants] has been particularly useful for me. I have also searched successfully using Miss [Surname] but have only used this with small local or regional papers or uncommon surnames to any useful effect.
    OCR techniques may improve over time but the two-pass problem may never really be overcome sufficiently.
    I’ve only just started using Elephind. I’ll have to check out their crowdsourcing options for papers I’m interested in. Like indexing for FamilySearch, crowdsourcing is such a great way to give back to the community and help others.

  2. Thanks, Rorey. Unfortunately – it is not OCR’s “fault” – when dealing with a less than optimal source such as old newspapers – there is not much that a “machine” or software can do automatically. Glad you have been successful.

  3. What great tips! I research for the family SHACKFORD. After reading this article, I searched on multiple OCR databases for SBACKFORD and found all sorts of newspaper articles that I hadn’t seen before. In come cases 20%-30% new articles!

  4. A couple of things I’ve learned was that when a few men are mentioned in one article, they’ll be listed a “Messrs Smith, Little and Baker”, rather than Mr Smith, Mr Little and Mr Baker. I’ve also found searching for the town or region along with a surname helps to narrow down your search.

  5. At the Fultonhistory.com site the OCR will occasionally read capital ” S ” as the number ” 8 “. Thus to find all references to A. S. Norton you will also have to search for A. 8. Norton. Other sites may have this idiosyncrasy also.

  6. I’m shocked no one has made search software that inherently knows letters that are confused with each other. For instance, searching for Carpenter will also bring up results for Carpenfer, Carponter, etc. Someone’s gonna make a killing if they write such a program.

  7. I think OCRed text searching could be improved by storing glyph (letter) elements individually, rather than ASCII codes for the recognized letters. This would allow searching based on what words
    look like. The glyph elements to be stored would be things like risers and danglers, for example. This system might consume slightly more memory for storage, but it would be worth it.

  8. What do I do about “Goth” being recognized as “60th”? I’m getting hundreds of hits with all sorts of “60th” – street, regiment, anniversary, etc…. 🙁

  9. I found this post to be quite helpful, so I would like to add to it. I’ve discovered that a capital M is often identified as Al or Ai. While searching for my Mercer ancestors, I tried Alercer and Aiercer, and got several relevant hits.

Leave a comment

Your email address will not be published. Required fields are marked *