Release of iqdb for e621 (Reverse Image Searching)

In category: e621 Tools and Applications

This is something I've been working on and tweaking for the past few days that I thought might be of some interest to people here. Basically similar to other reverse image searching tools such as Tineye, but should be much more accurate for e621 - among some other site-specific features. Should be very useful for finding dupe images, as well as finding higher-res copies of images you might have from other sites, and many other uses you may come up with.

Current features/info:
- Image/thumbnail/modified-image searching; should return the e621 page of the image you input.
- Upload directly from an image URL without needing to save to your disk.
- "mask explicit images" button, hiding images marked as explicit until you hover your mouse over them.
- Fast searching; typical image takes less than a second to query my database running on a mediocre server.
- Not heavy-weight; only does a few .xml request on e621 every few hours, and downloading a few thumbs from that at the same time (usually totaling less than a couple hundred kbs of traffic)
- idk some other cool/boring nerd shit.

Planned features:
- Its own favicon+theme (I just copied some of iqdb's and e621's css since I am lazy and bad at designing - so no one would be eyeraped by a black-on-white page)
- Your suggestions?

Link: http://iqdb.harry.lu/

Please post or PM me any comments/ideas/suggestions/bug reports/etc, thanks!

Dogenzaka
Privileged
5 years ago

Besides everything looking like it's zoomed out, i've been waiting for this. You should add a feature that lets us input an image's url from another site, so we don't have to save the image first.


1) Make it run for every upload, the uploader must then decide whether to save the image or delete it, if a very high similarity warning pops up. We will get fewer duplicates that way.

2) Make it compare every existing image on e621 with every other existing image with idle time, mark potential duplicates, let users fix it. How many years would it take?


51823611904 comparisons would be necessary to compare all existing images with each other.
If one comparison takes exactly 1 second (including writing the suspicous combinations in the database) and there is only one comparison at a time possible and there is no wait time between comparisons that would add up to:
599810 days =
1643 years

It would be 102 years with 8 parallel comparisons and 0.5 seconds for each comparison.

Time for a server upgrade. Or rent a scientific super computer.


Akkira said:
51823611904 comparisons would be necessary to compare all existing images with each other.
...
It would be 102 years with 8 parallel comparisons and 0.5 seconds for each comparison.

Time for a server upgrade. Or rent a scientific super computer.

It could just compare hashes to find exact matches, and a similar thing to a hash to find similar images. None of that would take 1 second per comparison.


Works pretty reliably. Does super zoom in things though

Kotep
Privileged
5 years ago

Akkira said:
nerdy math stuff

Or just find an algorithm that looks for basic matching characteristics between pictures. Like run all of the (gender) (species) tagged posts against each other, excluding ones marked as being from different artists.


Raiden_Gekkou said:
everything looking like it's zoomed out

You should add a feature that lets us input an image's url from another site, so we don't have to save the image first.

I don't have that issue. What browser are you using?

And yep URL upload is planned. For now, in most browsers, you can just put a URL into the file field and press enter, then after a second or so Windows downloads and caches it and rehosts it locally from your temp files.

Akkira said:
1) Make it run for every upload, the uploader must then decide whether to save the image or delete it, if a very high similarity warning pops up. We will get fewer duplicates that way.

2) Make it compare every existing image on e621 with every other existing image with idle time, mark potential duplicates, let users fix it. How many years would it take?

1) I am unsure what you mean. It's not explicitly designed for finding dupe images, nor does it access e621's API. There are also many images that have almost the exact same look as other images, but with slight mods (see https://e926.net/data/de/44/de44ec3ac080c4875f5c381f277545c5.png for example).

2) It wouldn't actually take that long (it usually takes less than a second for 32 different db queries on a server from 2003), but see point #1 :)

RedOctober
Privileged
5 years ago

I think Akkira is suggesting e621 implement this sort of thing on-site, to run every time someone uploads. If there is a super similar image going up, the uploader can either be like "oh yeah, totally the same image from a different place, nvm," or "It's an edit, so they're different, upload away," or "Oh, this one's higher res, I'll be sure to go flag the older post."

Dogenzaka
Privileged
5 years ago

yesthebest said:
I don't have that issue. What browser are you using?

I'm using Firefox, but the problem fixed itself.

ktkr
Former Staff
5 years ago
2013 :< animated cat cute english_text feline female fur grey_fur lol_comments looking_at_viewer low_res mammal pusheen pusheen_corp simple_background squigglevision text

Rating: Safe
Score: 136
User: snek-
Date: December 10, 2013

Thank you OP. This be useful.
A search bar on the result page will come in handy.


RedOctober said:
I think Akkira is suggesting e621 implement this sort of thing on-site, to run every time someone uploads. If there is a super similar image going up, the uploader can either be like "oh yeah, totally the same image from a different place, nvm," or "It's an edit, so they're different, upload away," or "Oh, this one's higher res, I'll be sure to go flag the older post."

Ah. That is a pretty useful idea. Though that would require the user still upload the file somewhere, since e621's upload preview just displays the image sourced from the user's local disk... so it wouldn't exactly be possible to automate like that without prompting every uploader every time. Could add it to e621's uploading-readme page, but it's not like most people read that :P.

Raiden_Gekkou said:
I'm using Firefox, but the problem fixed itself.

Yay! I'll just assume the css failed to load or something then.

ktkr said:
Thank you OP. This be useful.
A search bar on the result page will come in handy.

Can you elaborate what you mean on this?


yesthebest said:
Can you elaborate what you mean on this?

No idea, but it probably have to deal with putting forth more results if the image search didn't really find the exact image of what the user submitted in.

In other words, put a search bar just in case if the image search didn't get the image or find the image on e621 and the image is there. I hope this makes sense.

Derp.

ktkr
Former Staff
5 years ago
2013 :< animated cat cute english_text feline female fur grey_fur lol_comments looking_at_viewer low_res mammal pusheen pusheen_corp simple_background squigglevision text

Rating: Safe
Score: 136
User: snek-
Date: December 10, 2013

yesthebest said:
Can you elaborate what you mean on this?

Having the search bar from main page also on the result page so one doesn't have to keep clicking back to search for something new.

Char
Former Staff
5 years ago
alpha_channel ambiguous_gender charmander chibi edit fire flaming_tail full-length_portrait humor looking_at_viewer low_res nintendo orange_skin pokémon pokémon_(species) portrait simple_background solo transparent_background unknown_artist video_games

Rating: Safe
Score: 15
User: Halite
Date: June 23, 2013

RedOctober said:
I think Akkira is suggesting e621 implement this sort of thing on-site, to run every time someone uploads. If there is a super similar image going up, the uploader can either be like "oh yeah, totally the same image from a different place, nvm," or "It's an edit, so they're different, upload away," or "Oh, this one's higher res, I'll be sure to go flag the older post."

Believe me, that is something I've been badly wanting ever since I started administrating the site 2 years ago.

Dogenzaka
Privileged
5 years ago

I mentioned this is the feature request thread long ago, but I guess nothing came of it. Just glad to see it's happening now, even though I hardly upload anymore.


speak to me in pms please


ktkr said:
Having the search bar from main page also on the result page so one doesn't have to keep clicking back to search for something new.

Aha, gotcha. Should be there now.


This is incredibly useful. Thank you for this.


No, no, no, no!
I am not pleased at all.

oh noes, can not connect to database at Wed, 24 Oct 2012 14:19:35 +0000 ;_;


Akkira said:
No, no, no, no!
I am not pleased at all.

oh noes, can not connect to database at Wed, 24 Oct 2012 14:19:35 +0000 ;_;

Unlucky! I was running some high-intensity stuff on the server for like an hour at that time without modifying its priority, so the searching probably took too long so it timed out, hah. It should work fine normally (and works now; just tested it).

Granberia
Contributor
5 years ago
2015 animal_humanoid apron blush clothing dragon dragon_humanoid female granberia green_scales hair hi_res humanoid looking_at_viewer monster_girl monster_girl_(genre) monster_girl_quest naked_apron red_hair scales simple_background solo standing tau_1111 video_games yellow_eyes

Rating: Safe
Score: 15
User: Granberia
Date: September 02, 2017

Is it just me, or something has broken and search never finds relevant pictures? I downloaded and tried to search for several random pictures, and it didn't find any.


Update!111111
- Implemented URL searching; paste in direct URL to image and have search.

Gilda_The_Gryphon said:
Is it just me, or something has broken and search never finds relevant pictures? I downloaded and tried to search for several random pictures, and it didn't find any.

Link what pictures don't work for you if you're sure they are on e621 and gives no results. Also sometimes my database updater got hung up a few times, probably what happened at that time.


I think this should be promoted much more.


Der_Traubenfuchs said:
I think this should be promoted much more.

I agree!

Also people are free to lightly use http://iqdb.harry.lu/?url=<ImgThumb> with some URL to see results that way (maybe a bit easier than automating a POST), but I don't plan to make some json API or anything like that.


yesthebest said:
I agree!

Also people are free to lightly use http://iqdb.harry.lu/?url=<ImgThumb> with some URL to see results that way (maybe a bit easier than automating a POST), but I don't plan to make some json API or anything like that.

You are making me all hot and sweaty.

Loading (? KB)... failed


Der_Traubenfuchs said:
You are making me all hot and sweaty.

Loading (? KB)... failed

Should work, what's the URL you're loading? Make sure you don't have the <>'s around the URL as well.


yesthebest said:
Should work, what's the URL you're loading? Make sure you don't have the <>'s around the URL as well.

Don't worry, I bet he just clicked the sample URL you provided, without replacing the placeholder.

Also, I've already wanted to post this one but I kept forgetting it. There's this bookmarklet that can be used for easy image uploading from another website. So I made one for myself to be used with your iqdb.
Simply put this in a bookmark:

javascript:location.href='http://iqdb.harry.lu/?url='+encodeURIComponent(location.href)

It can be used the exact same way.


blackest_vulture said (forum #136151):
I'm having trouble using the Reverse harry.lu Search since yesterday. It takes minutes to search for an image instead of seconds.

The strange thing is that the engine still says it searched the image in half a second.


EsalRider said:
blackest_vulture said (forum #136151):
I'm having trouble using the Reverse harry.lu Search since yesterday. It takes minutes to search for an image instead of seconds.

The strange thing is that the engine still says it searched the image in half a second.

Thanks for the quoting :) Too bad we can't vote on forum posts. Please consider my symbolical +1.