As a film archivist I wanted to learn more about possible content based image retrieval (CBIR) tools, to evaluate if they could be a solution for the extraction of automated metadata for the videos in the collections I work with.
2. Select the video tab on the left side.
5. Under filters, click on Scene filter.
Paste previously copied directory path.
Running one screenshot at a time, I found that these reverse image-search applications’ have a greater success rate if the inputted image is in color, and often return non useful results for black and white images. The applications seem to have trouble discerning the content from the images itself with B&W.
The development of a CBIR tool from scratch goes beyond my personal skill level. So I attempted to develop a rudimentary workflow to carry out this process.
Finding a semi-automated way to input video into reverse image search tools.
Automatic screenshots
Tineye and Google images are the most popular reverse image search engines. But they only allow for still images.
VLC allows for automatic screenshot capturing at specific time intervals.
1. Preferences (⌘,)
click Show All
Specify snapshot directory and copy it.
3. Under the video tab, select the filter tab.
4. Check scene video filter.
Paste previously copied directory path.
Set the recording ratio. This is the frame interval between automatic screenshots. For a 30fps video, 300 fps ratio would screenshot every 10 seconds.
Results:
Batch Reverse Image Search
When I started researching this project I had found BRIS Tool (Batch Reverse Image Search Tool) developed by Ronald Kwan in 2015.
Using GitHub's codespaces I was able to run the code.
But the program was no longer running.
R.I.P BRIS Tool
2015 – 2022
Why? (Not sure)
Possibly due to changes in Google’s API.
Phasing out of Google's classic reverse image search.
Incorporation of Google Lens.
Hope of establishing a semi-automated way to do this workflow was lost...
Creating an application that carries out batch reverse image searches on a platform such as Google Lens, would require further research on their API.
Further thoughts
Google lens incorporates a variety of search tools, such as optical character recognition, translation, and greater communication with Google shopping listings.
The bundling of these search tools has better returns than 1:1 reverse image matches.
Google Lens:
TinEye:
Running one screenshot at a time, I found that these reverse image-search applications’ have a greater success rate if the inputted image is in color, and often return non useful results for black and white images. The applications seem to have trouble discerning the content from the images itself with B&W.
This issue restricts the possible applications set out in the user case, as audiovisual collections are usually not made up of a single film process.
Further Thoughts p.2
After scouring the contentious AI "enhancing" community I found that before running colorization neural networks such as DeOldify they tend to apply De Noise and sharpening processes which essentially remove the film grain of the digitized element.
Even though carrying out these processes prior to AI analysis, supposedly returns better results, their application to archival collections is at odds with the preservation of the material history of the physical object.
Conclusion
Although the development of a functioning workflow to carry out a semi-automated reverse image search from video was a failure, the project allowed me to delve a little bit deeper into the topic of computer vision, APIs and CBIR applications.
Comments
Post a Comment