Keito
Member
1 year ago

e621 updater - tag local files and download favorites!

In category: e621 Tools and Applications

I'm a little addicted to tagging my porn, and after a few months of tagging my yiff folder contents i've realised i can make my job easier.
This program is made to search e621.net for tags and add them to the images it founds.
Be prepared for a storm of trash (1) tags and typos. On the other hand, you can use the results to see badly tagged images and fix them here on e621.

What can my tool do:

  • Download favorites for any user (by username)
  • Add tags from e621 to your saved pictures
  • Download updated pictures if they were replaced there on e621
  • Find and favorite e621 posts of your locally saved pictures

--------------------
DISCLAMER: The following procedure is highly stupid. I'm not accountable for any injuries. Impalement or circumcision may occur.
If you're running this for the first time, i would advise you to make a copy of your gallery first.
Never not backup your shit.
--------------------
Current version is v9.2 (v5 GUI) (01.08.2017)

Dropbox
MEGA

CURL.exe for sync mode

Screenshot
Speed comparison for different settings

It uses 1 additional program: exiftool.exe. See readme for more info on the program.
--------------------
How to use:
1) Read readme first. No, really. If you're too lazy to do that at least make a backup of your images.
2) Unzip and place all files into (any) desired directory
3) Run e621updater.exe
4) Choose your images folder
5) Decide if you want to update pictures, tag them or download favorites (you better update pictures before tagging them)
6) Go to sleep
--------------------
Please note that "Shutdown after" requires admin rights. If you want to use the feature, run this tool as admin or enable that option first after tool start (it will restart itself to get admin rights).

It will run through all the PNG and JP(E)G images it founds and try to tag them.
It will take less than 2 seconds to tag one image.
Last time i've tagged 14725 images it took me 4 hours 18 minutes. Plus 2 hours 18 minutes for updater.

Please have mercy on my tool and remove any occurrences of multiple extensions (e.g. ass.jpg.png).

▼ Questions and Answers

Q: Will it tag everything?
A: Everything that is found on e621 by searching md5 hash. No, i doubt it will tag ALL of your files.

Q: Will it break everything?
A: Probably not. In the worst scenario, it will remove present metadata and waste your time. But you better read README and do as it says.

Q: "Updater mode"?
A: TL;DR: it will download better versions of your saved pictures if there are any, moving old ones into "backup" directory.

Q: "Downloader mode"?
A: Select your images folder upon start, enter your username, select needed ratings. If you did everything right, it will get a list of your favs, remove existing files from queue and download everything that's not skipped.

Q: "Sync mode"?
A: Select your images folder upon start, enter your username and API key. If you did everything right, it should get a list of your saved images and add them to favorites on e621.

Q: Where can i find my API key?
A: You should generate it there: https://e926.net/user/api_key

Q: "More information"?
A: Don't enable it unless you are bored to death and want to see some information that you will never need. It will also slow the process a bit. I don't even know why i wasted my time to add it.

Q: "Get MD5"?
A: It's a feature i've added some time ago. I don't use it and i'm not even sure it still works. I really hope that you are not using it too and keep your files named after their MD5's.

Q: WIll it tag GIFs?
A: It won't. They are bad with metadata. Only PNG, JPG and JPEG files are supported.
WEBMs and SWFs are supported in sync mode.

Q: What if %filename% is not found on e621?
A: Nothing will happen, that file will be moved to "NotFound" directory.

Q: How much files can i tag?
A: The number of files is unlimited. Tag all the things.

Q: How exactly is it tagging my images?
A: Using XMP dc Subject for tags and XMP dc Creator for artist. It is compatible with Adobe, Windows and Picasa. Probably with all the others.

Q: Some of my images are already tagged. What's going to happen?
A: All of your tags are going to be erased. Sorry.

Q: Can i use that with proxy?
A: This program will use a proxy server to access the Internet if such a proxy has been configured in Microsoft Internet Explorer's settings (not applicable to WGET edition).

Q: Are you using API?
A: Yes!

Q: What if my internet connection fails in progress?
A: It will make a few attempts to restart and will automatically pause after 3 failed attempts (~5 minutes), unless you're using "Disable network checks" option. If you use it, all files will be moved to NotFound directory. Some modes are not optimized for bad network state, so they will throw an error and stop working. Favorites and sync modes are affected by that problem.

Q: Is it a virus?
A: No. Current version is not a virus. I may consider adding it in future versions.

▼ README.TXT contents

e621 tagger v9.2 (GUI v5)
This program is made to search e621.net for tags and add them to the images it founds.
See "use cases" at the end of the file.
It is expected that your files are named after their md5 checksum.
It is the default name when you save file manually, but if you use grabbers, they may save files using post ID.
If your file names are not their md5's, you want to check "Get MD5" before starting the proccess.
If your files are named like "8bcfd910f5218a8fe29ba8df6b61854f", they are named as expected.
If the file was changed since you saved it (e.g. better version uploaded), the md5 name on server will change and tagger won't find the new image. It will get the old info about your specific image.
To avoid that, download your favorites, then remove duplicates (you want to remove older versions) using ImageDupeless.
You can also use the "Updater mode" option. It is designed specifically to fix that problem.

It will only tag PNG and JP(E)G images using XMP. GIFs or any other extensions will be ignored.
If you have metadata in your images, it is going to be removed for compatibility. It includes, but not limited to: TAGS and RATINGS.
There is no way to avoid this in the current version.

To avoid tagging everything over and over again as you save new files, do one full run with this tool, then save your pictures into some temp folder and run this tool from there before moving them to your main storage.
Download pictures, tag them, move to your main folder. You don't want to spend a few hours every time you save one picture.

This tool will often create and remove temp files to work. I can't say running this on SSD drive is a good idea.
On the other hand, storing a ton of your porn images on SSD is a bad idea too.

Files:
exiftool.exe - add or remove metadata
http://www.sno.phy.queensu.ca/~phil/exiftool/

curl.exe - required for sync mode to work, otherwise not needed
https://curl.haxx.se/

I've only tried that on my own PC running Windows 10. If you use some weird shit, everything may fuck up.
I would adivse you to try it on a few files to see how it goes before raping your fap directory:
Put this tool into separate directory and place some test subjects nearby. When it finishes, check the results.
If you have any problems report them to a forum thread: https://e926.net/forum/show/172541

This program will process one file in less than 2 seconds. If you have a lot of pictures, you'll have a lot of time to think about your life.
You may want to leave it running at night instead of babysitting the process. Don't forget to disable hibernation, sleep and other helpful stuff before starting.

If you're going to use Adobe Lightroom, you will suffer from the amount of tags.
If you don't like pain, use Picasa or a fucking Windows Explorer.
If you do (i do), i made this tool Adobe-compatible. Enjoy your tagging.

Changelog:
9.2 QoL
- App is x32 now, should work for everyone.
- Sync mode bugfixes:
-- SWFs and WEBMs are correctly favorited.
-- MD5 is now correctly found for SWFs and WEBMs.
- GUI bugfixes.
- Choose Another Folder remembers your last folder and starts from there.
- "Get API" button added for sync mode.
- DLLCall added to prevent PC from sleeping while app is working.
-- Can be disabled by modifying config file: manually set PCAwake to 0.

9.1 Sync
- No longer limited to images: webm and flash support added.
- Tool will calcualte MD5 for all incorrectly named files.
-- It's still not going to work if you've changed their original MD5 by any means (including tagger!).
- Config file added (e621updater.ini).
-- Settings saved for now: working directory, username and API key, "network checks" and "backup" checkboxes state.
- Updater mode backup is fixed

9 Sync
- Sync mode added!
-- Enter your username into the "name" field (get it from https://e926.net/user/home to be sure).
-- Generate an API key and enter it into the "API key" field (get it from https://e926.net/user/api_key ).
-- Press start and all local images will have their E621 posts favorited.
-- No checks are coded: make sure there are no mistakes and spaces in both fields or you'll simply waste your time.
-- Sync mode requires curl.exe to work. Proxy is not supported.
- Small GUI changes.

8 Favorites
- Favorites downloader added!
-- Just select needed mode, enter your username and it will get all the favs you have.
-- Ratings selection. Able to download any combinations of ratings (e.g. only E and Q, without safe images).
-- Existing files (that you already have in target folder) are skipped.
- "Tag after updating" option added for favorites and updater modes.
- "Fast mode" is now always on, option removed. No drawbacks from using it.
- Now calculating MD5 inside the tool itself.
-- Removed MD5.exe
- GUI changed.
-- More information checkbox added for all modes.
-- Information field on bottom of the tool is replaced with list box. Tool will probably work too fast for you to get any info from there.
- Optimized JSON parsing.
- Corrected mistakes in log file again.

7 Need for Speed - v3
- Performance improved
- 2 features added: "Fast mode" and "Disable network checks"
-- "Fast mode" disables waiting for program to finish to in-app calls (for all exif and some wget calls). Should not cause problems. Experimental, default on. ~50% performance increase.
-- "Disable network checks" disables... Network checks. All of them. Don't use it if your connection is unstable (VPN, proxy, shitty network, etc). ~50% performance increase.
-- Use both for maximum speed! Almost 3x times faster.
- Small GUI changes
- Error in log file for updater is fixed

6 GUI v2
- WGET and it's libraries are removed.
-- Internet Explorer 3 or greater must be installed for this program to work. Firewalls or the presence of multiple network adapters may cause this program to fail. You can use WGET edition (see bottom of this post) if you have problems with that.
-- You can now use proxies! This program will use a proxy server to access the Internet if such a proxy has been configured in Microsoft Internet Explorer's settings.
-- No longer outputs -user-agent=Keito/Tagtool1.0 for e621 admins. Sorry guys. Can't figure out how to restore this without WGET.
- Added "Shutdown after" checkbox. Will shutdown your PC when proccess finishes. Useful!
- "Force remove old tags" checkbox is no longer checked by default. Should make the whole proccess faster. (1-2 seconds per image)
- Added "time elapsed" and "time left" counters. Time left is very inaccurate as expected. They update after every image processed (not realtime).
- ImagesList file is now correctly deleted when program closes.
- Creating after_log.txt file after every launch. Might be useful for everyone who used "Shutdown after" and wants to know how proccess went after.
-- Only basic statistics for now. Time start, time end, files processed, files skipped.
- Minor GUI changes.

5.1 Bugfixes
- "Close app" question at the end of the proccess is fixed.
- Window no longer duplicate if answered "No" after "Update" sequence
- App will correctly restart if answered "No" after "Tag" sequence

5 GUI
- GUI v1
- Batch script is recoded into a proper EXE with GUI. It's slightly less miserable now!
- Tagger and updater consolidated. Use the program and choose what you want to do.
- xidel is removed, wget added instead.

4.2 Updater filenames
- Updater will now calculate md5 if file is not named after the checksum and try to use it for search.

4.1 Bugfixes
- I've removed filenames checks completely, since it works way too slow for me (and probably for someone\everyone else). I don't expect everything to stop working because of that, but feel free to report any bugs.
- Added some more comments to both updater and tagger.
- No longer deletes old log after launching the tagger.

4 Updater
- Updater is now included. You may think that it updates this tool, but it will download new pictures instead of the old ones saved locally.
- Many pictures may have new higher resolution, not watermarked, not fucked up version uploaded to e621. This tool will find all the new pictures and download them, moving your old files to "backup" folder.
- Still requires testing, since technically it will save parent of deleted picture. It may be the same picture, or it may be edited one (still i guess it should be better if the old one got removed)
- Only works with md5 filenames at the moment.

- Possible fix to authors: it should now get authors for not-md5 filenames too.

3.2 Authors! Again.
- I've moved code that is getting artists from e621 to the external file "ARTISTS.bat"
- Now getting all the artists! For some reason, Adobe recognises n-1 artists (not less than 1 tho). Fuck Adobe!
- "unknown_artist" is no longer an artist.
- "avoid_posting" is no longer an artist.
3.1 Filenames and authors again
- "conditional_dnp" is no longer an artist.
- If you encounter any more trash author tags, let me know.
- All invalid filenames should now be ignored.
3.0 Filenames
- Moved off from "md5 only"
- Tool will remove all dots and underlines from all filenames in directory. Not able to remove other invalid sybols yet.
- Tool will now calculate md5 if file is not named after the checksum and try to use it for search.
- Added PULLER.bat and md5.exe to implement that. Still not a virus.
- Will now save author to the "creator" field as well as into tags!
- Keeping it in tags just in case someone's software only works with one field, not with both tags and authors. For now.
- Will save author to xmp-dc:creator
- Removed excessive -api PNGEarlyXMP when calling exiftool. We don't need that for JPG files. We still do need that for PNGs, for some reason.
2.0 API
- Reworked the batch file to use JSON.
- Now 2x faster!
- Getting tags even for deleted pictures.
- Fixed possible PNG bug (not writing anything to PNG)

1.0 Release

DISCLAMER: The following procedure is highly stupid. I'm not accountable for any injuries. Impalement or circumcision may occur.
If you're running this for the first time, i would advise you to make a copy of your gallery first.
Never not backup your shit.

Here's what's going to happen if you go with the default settings (Get tags, MD5 unchecked): all of your "myfavoritepicture.jpeg" will be ignored. This program will only work with files that are named like random amount of numbers and letters (00d5b72e6dac8cbf80196f3d9fc3c46e.jpg for example). All of them will be tagged, nothing bad is going to happen.
If you check the "Get MD5" box you will tag more pictures, "myfavoritepicture" will most likely be tagged too. You will have to choose if you want to rename all the files after their MD5 checksum or not. If you press NO, you will never be able to tag "myfavoritepicture" with updated tags again: it will never find a file with the changed checksum (you've tagged it, remember?). If you press YES, you will never see "myfavoritepicture" in names again, but all of your pictures will be updated and tagged correctly in future.

USE CASES:
1) I have a lot of pictures in one folder and i have no idea how they are named.
Recommended settings: Use "Updater mode" function, then use "Tagger mode" with "Get MD5" checkbox ON (choose YES when asked about renaming).

2) I have a lot of pictures and all of them are named correctly. Yes, i know what "correctly" means!
Recommended settings: Use "Updater mode" function, then use "Tagger mode" with "Get MD5" checkbox off.

3) I have a lot of pictures and i've put a lot of effort into renaming them. Keep my filenames!
Recommended settings: Use "Tagger mode" with "Get MD5" checkbox ON (choose NO when asked about renaming).

4) Images not tagged after any of your suggestions before!
Recommended settings: Enable "Force remove old tags" checkbox, use "Tagger mode" with "Get MD5" checkbox ON (choose YES when asked about renaming).

▼ Planned improvements
  • Bugfix: spaces in the working folder path break tagging.
  • Improve sync mode (at least add proxy support and\or make it show skipped images).
  • Add network checks to all modes.
  • Expand config functionality.
  • Recovery mode: Find external sources for deleted images (FA and stuff?).
  • Performance optimizations:
    • Get rid of as much temp files as possible.
    • Look into possible optimizations of exiftool calls (batch tagging).
  • Alternative tagging methods to avoid modifying files (and to keep actual MD5).
    • None of available options look good for me. On ice.
▼ Changelog
  • 9.2 QoL
    • App is x32 now, should work for everyone.
    • Sync mode bugfixes:
      • SWFs and WEBMs are correctly favorited.
      • MD5 is now correctly found for SWFs and WEBMs.
    • GUI bugfixes.
    • Choose Another Folder remembers your last folder and starts from there.
    • "Get API" button added for sync mode.
    • DLLCall added to prevent PC from sleeping while app is working.
      • Can be disabled by modifying config file: manually set PCAwake to 0.
  • 9.1 Sync and config
    • Sync mode updated:
      • No longer limited to images: webm, gif and swf support added.
      • Will calcualte MD5 for all incorrectly named files.
    • Config file added (e621updater.ini):
      • Settings saved for now: last used working directory, username and API key, "network checks" and "backup" checkboxes state.
    • Updater mode backup is fixed
  • 9 Sync
    • Sync mode added:
      • Enter your username into the "name" field (get it from https://e926.net/user/home to be sure)
      • Generate an API key and enter it into the "API key" field (get it from https://e926.net/user/api_key )
      • Press start and all local images will have their E621 posts favorited.
      • No checks are coded: make sure there are no mistakes and spaces in both fields or you'll simply waste your time.
      • Sync mode requires curl.exe to work. Proxy is not supported.
    • Small GUI changes
    • Updater mode backup is fixed
  • 8 Favorites
    • Favorites downloader added:
      • Just select needed mode, enter your username and it will get all the favs you have.
      • Ratings selection. Able to download any combinations of ratings (e.g. only E and Q, without safe images).
      • Existing files (that you already have in target folder) are skipped.
      • Connection checks are not supported at the moment. Don't use it for shitty network.
    • Now calculating MD5 inside the tool itself.
      • Removed MD5.exe
    • GUI changed.
      • "More information" checkbox added for all modes.
      • Information field on bottom of the tool is replaced with list box. Tool will probably work too fast for you to get any info from there.
    • "Tag after updating" option added for favorites and updater modes.
    • "Fast mode" is now always on, option removed. No drawbacks from using it.
    • Optimized JSON parsing.
    • Corrected mistakes in log file again.
  • 7 Need for Speed - v3
    • Performance improved
    • 2 features added: "Fast mode" and "Disable network checks"
      • "Fast mode" disables waiting for program to finish to in-app calls (for all exif and some wget calls). Should not cause problems. Experimental, default on. ~50% performance increase.
      • "Disable network checks" disables... Network checks. All of them. Don't use it if your connection is unstable (VPN, proxy, shitty network, etc). ~50% performance increase.
      • Use both for maximum speed! Almost 3x times faster.
    • Small GUI changes
    • Error in log file for updater is fixed
  • 6 GUI v2
    • WGET and it's libraries are removed.
      • Internet Explorer 3 or greater must be installed for this program to work. Firewalls or the presence of multiple network adapters may cause this program to fail. You can use WGET edition (see bottom of this post) if you have problems with that.
      • You can now use proxies! This program will use a proxy server to access the Internet if such a proxy has been configured in Microsoft Internet Explorer's settings.
      • 70-90% faster.
      • No longer outputs -user-agent=Keito/Tagtool1.0 for e621 admins. Sorry guys. Can't figure out how to restore this without WGET.
    • Added "Shutdown after" checkbox. Will shutdown your PC when proccess finishes. Useful!
    • "Force remove old tags" checkbox is no longer checked by default. Should make the whole proccess faster. (1-2 seconds per image)
    • Added "time elapsed" and "time left" counters. Time left is very inaccurate as expected. They update after every image processed (not realtime).
    • ImagesList file is now correctly deleted when program closes.
    • Creating after_log.txt file after every launch. Might be useful for everyone who used "Shutdown after" and wants to know how proccess went after.
      • Only basic statistics for now. Time start, time end, files processed, files skipped.
    • Minor GUI changes.
  • 5.1 Bugfixes
    • "Close app" question at the end of the proccess is fixed:
      • Window no longer duplicate if answered "No" after "Update" sequence
      • App will correctly restart if answered "No" after "Tag" sequence
  • 5 GUI
    • GUI v1
    • Batch script is recoded into a proper EXE with GUI. It's slightly less miserable now!
      • Still not a virus.
    • Tagger and updater consolidated. Use the program and choose what you want to do.
    • xidel is removed, wget added instead.
  • 4.2 Updater filenames
    • Updater will now calculate md5 if file is not named after the checksum and try to use it for search.
  • 4.1 Bugfixes
    • Removed filenames checks completely, since it works way too slow for me (and probably for someone\everyone else).
      • I don't expect everything to stop working because of that, but feel free to report any bugs.
    • Added some more comments to both updater and tagger.
    • No longer deletes old log after launching the tagger.
  • 4 Updater
    • Updater is now included. You may think that it updates this tool, but it will download new pictures instead of the old ones saved locally.
      • Many pictures may have new higher resolution, not watermarked, not fucked up version uploaded to e621. This tool will find all the new pictures and download them, moving your old files to "backup" folder.
      • Still requires testing, since technically it will save parent of deleted picture. It may be the same picture, or it may be edited one (still i guess it should be better if the old one got removed)
      • Only works with md5 filenames at the moment.
    • Possible fix to authors: it should now get authors for not-md5 filenames too.
  • 3.2 Authors! Again.
    • I've moved code that is getting artists from e621 to the external file "ARTISTS.bat"
      • Now getting all the artists! For some reason, Adobe recognises n-1 artists (not less than 1 tho). Fuck Adobe!
    • "unknown_artist" is no longer an artist.
    • "avoid_posting" is no longer an artist.
  • 3.1 Filenames and authors again
    • "conditional_dnp" is no longer an artist.
      • If you encounter any more trash author tags, let me know.
    • All invalid filenames should now be ignored.
  • 3.0 Filenames and authors
    • Moved off from "md5 only"
      • Tool will remove all dots and underlines from all filenames in directory. Not able to remove other invalid sybols yet.
      • Tool will now calculate md5 if file is not named after the checksum and try to use it for search.
      • Added PULLER.bat and md5.exe to implement that. Still not a virus.
    • Will now save author to the "creator" field as well as into tags!
      • Keeping it in tags just in case someone's software only works with one field, not with both tags and authors. For now.
      • Will save author to xmp-dc:creator
    • Removed excessive -api PNGEarlyXMP when calling exiftool. We don't need that for JPG files. We still do need that for PNGs, for some reason.
  • 2.0 API
    • Reworked the batch file to use JSON.
      • Now 2x faster!
      • Getting tags for pictures that was deleted.
    • Fixed possible PNG bug (not writing anything to PNG)
    • Updated exiftool to the most recent version. Just in case.

    1.0 Release

    ▼ What options to use

    1) I have a lot of pictures in one folder and i have no idea how they are named.
    Recommended settings: Use "Updater mode" function, then use "Tagger mode" with "Get MD5" checkbox ON (choose YES when asked about renaming).

    2) I have a lot of pictures and all of them are named correctly. Yes, i know what "correctly" means!
    Recommended settings: Use "Updater mode" function, then use "Tagger mode" with "Get MD5" checkbox off.

    3) I have a lot of pictures and i've put a lot of effort into renaming them. Keep my filenames!
    Recommended settings: Use "Tagger mode" with "Get MD5" checkbox ON (choose NO when asked about renaming).

    4) Images not tagged after any of your suggestions before!
    Recommended settings: Enable "Force remove old tags" checkbox, use "Tagger mode" with "Get MD5" checkbox ON (choose YES when asked about renaming).

    ▼ Other versions

    Previous (v8) version:
    https://dl.dropboxusercontent.com/u/21291309/updaterv8.zip
    https://mega.nz/#!PZsSyJrT!EMihNFssojXmJmC_AtvkUkbq82r7aahfDW3CUcCYcKM

    Obsolete WGET edition:
    WGET edition is no longer updated.
    It's at least 2 times slower than the default version on top of this page. I'm too lazy to code a favorites downloader for it.
    Maybe i will update it in future, but not at the moment.
    Make sure you've seen this before downloading WGET version

    Current version is v7 (v3 GUI) (06.10.2016)
    Dropbox: https://dl.dropboxusercontent.com/u/21291309/updaterv7wget.zip
    MEGA: https://mega.nz/#!CYtH2ajR!77cCXCxOxxj54Z84xJhrow_5ad3oZ1GnENnlXmJRjIs

    Obsolete batch:
    These batch files are not working and i'm not updating them anymore.
    Last time i've looked through them they needed user agent parameter added to all xidel.exe calls to get them working.
    Have fun!
    Dropbox: https://dl.dropboxusercontent.com/u/21291309/taggerv4p2.zip
    MEGA: https://mega.nz/#!jd0ySKDa!TB-00GGFmqHkKHYgqBLBSEi16bE1BlgVbNyoI2P7Pgc


    I haven't actually tested this, but I wanted to mention a couple things.

    Keito said:

    IT ONLY WORKS FOR IMAGES THAT ARE NAMED AFTER THEIR MD5 CHECKSUM. NO TAGS FOR YOUR "yomamma.jpg" IMAGES!

    The md5 is hard to rely on for this (particularly for images that have added metadata), but why not attempt to calculate md5 when the filename match fails?

    I know certUtil -hashfile somefilename MD5 works for a single file, but you might be able to push your input through that and get something meaningful out of it (I haven't tried doing this). Else I'm sure there's a freeware utility out there that can do this for you.

    Secondly, I highly recommend using our API to retrieve the tag information. It should make queries run noticeably faster but mostly it will be much much easier for you to parse and less likely to break than trying to scan the page for strings.

    I prefer JSON myself, but we support XML as well.

    JSON: https://e926.net/post/tags.json?md5=b727f5f7982e8d6d1c6e01754562c075
    XML: https://e926.net/post/tags.xml?md5=b727f5f7982e8d6d1c6e01754562c075

    Just note that the tag types are actually written incorrectly on the API help page (I haven't gotten around to updating it yet). These are the correct ones:

    "General"   => 0
    "Artist"    => 1
    "Copyright" => 3
    "Character" => 4
    "Species"   => 5
    

    Keito said:
    If you can tell me how to implement API, i'd do that.

    I don't personally tag my local files, but I figured I'd chime in on this.

    If you can find yourself a JSON parser or XML parser to replace your HTML parser, using the API is quite simple. Just replace your request URL https://e621.net/post/index/1/md5: with https://e621.net/post/tags.json?md5= or https://e621.net/post/tags.xml?md5= and make whatever small changes are needed to tell the parser to give you just the tag names from that.

    Here's an example of the output:
    https://e926.net/post/tags.json?md5=466A53E868164275F9386F0A90A3FE8F
    https://e926.net/post/tags.xml?md5=466A53E868164275F9386F0A90A3FE8F

    JSON is slightly more efficient than XML, but either should give your program a significant speed boost.

    Here's a link to the full API documentation if you wish to learn more: e621:api

    Cheers. :)

    edit: whoops, parasprite beat me to it.

    Keito
    Member
    1 year ago

    Thank you for the ideas.
    I will switch to API later then, since xidel supports XML (maybe JSON too?). It'll be quite hard for me since API is giving a lot of useless (for what i'm doing) info. I literally only need tags as a plain text, delimited by ",". I'm not even parsing the actual post page, but the search results page tooltips, because they are closer to what i want there. :)
    See: http://c2n.me/3nX66OM.png
    Parsing that XML page actually look like a lot of more work from the first look. Okay it's actually not that bad after i'll figure out "properties" i think...
    But i like the fact that i can export artist tag into appropriate XMP tag. Without tag types, that was impossible.

    Regarding filenames, i'm not sure what else can i rely on to automate the process. MD5 sounds like a best shot.
    And if the md5 has changed on the server side, there is no way to find a new image automatically AFAIK.

    Not sure why would anyone rename his files, but i'll consider adding MD5 checks later.


    Keito said:
    Not sure why would anyone rename his files, but i'll consider adding MD5 checks later.

    Well, I like the filename to be actually meaningful -- much better than '6a7931cc3867dbaced4c032b4af79888', and calculating md5sum only takes a few seconds per 10mb of input, which is completely trivial for 99% of files.

    I suppose I should publish my own tool to do this.

    BTW, if md5 has changed on the server side, that means that the post you are looking for still exists, as a deleted post (probably with a link to the revised post in the deletion reason.).

    Keito
    Member
    1 year ago

    Funny thing it indeed exists. I was wondering how did i manage to break "if file exist on server" checks and just now realised he has found all the files when using JSON.
    I will contact you later if i fail to found a tool for that.
    There is a workaround to first get the original post id, then go to the actual page and parse HTML... But that's not worth it since most of the results i've checked are deleted permanently (takedown requests and stuff).
    I wonder if we can add reason to the API?

    Keito
    Member
    1 year ago

    Updated to v2.

    • Reworked the batch file to use JSON.
      • Now 2x faster!
      • Getting tags for pictures that was deleted.
    • Fixed possible PNG bug (not writing anything to PNG)
    • Updated exiftool to the most recent version. Just in case.

    Forgot to update readme before updating. *sigh*

    Chessax
    Contributor
    1 year ago
    2010 ambiguous_gender black_lips canine cute daffodil_(flower) day eyes_closed feral flower forest fox fur half-length_portrait inner_ear_fluff long_mouth mammal markings matt_binstead nature orange_fur outside photo plant portrait real red_fox side_view sniffing snout solo tree whiskers white_fur

Rating: Safe
Score: 219
User: zoodno
Date: September 17, 2010

    I don't like internal custom metadata that is not based on intrinsic properties because I hate multiple file versions. So if you use this tool, please do not redistribute your files, but who am I to tell you what to do with them... *grumbles*

    Otherwise, go crazy! :)

    Keito
    Member
    1 year ago

    Chessax said:

    Yep, i agree with that. If you guys use this tool, you better don't upload your files anywhere, since they are not original images anymore.
    They will differ at least by size and md5 checksum, so we may end up with 10 similar pictures with different tags.


    Ah, now I understand why the files must be named in accordance with md5sum. My tool avoids that problem by not modifying the file at all, using TMSU to tag it instead. This has corresponding tradeoffs, of course (if your file manager isn't TMSU aware, renaming or duplicating the file after you've tagged it is not a good idea. On the other hand, you can tag any file type, not just PNGs or JPGs.)

    However, previously I had some files which included both a normal name and md5sum string, separated by a comma. That is an option you could consider supporting.

    Keito
    Member
    1 year ago

    We're version 3 now (and a quick 3.1 update!)
    I had little time to test it, but it should work okay.

  • 3.1 Filenames and authors again
    • "conditional_dnp" is no longer an artist.
      • If you encounter any more trash author tags, let me know.
    • All invalid filenames should now be ignored.
  • 3.0 Filenames and authors
    • Moved off from "md5 only"
      • Tool will remove all dots and underlines from all filenames in directory. Not able to remove other invalid sybols yet.
      • Tool will now calculate md5 if file is not named after the checksum and try to use it for search.
      • Added PULLER.bat and md5.exe to implement that. Still not a virus.
    • Will now save author to the "creator" (author in windows) field as well as into tags!
      • Keeping it in tags just in case someone's software only works with one field, not with both tags and authors. For now.
      • Will save author to xmp-dc:creator
      • Only 1 artist per picture at the moment.
    • Removed excessive -api PNGEarlyXMP when calling exiftool. We don't need that for JPG files. We still do need that for PNGs, for some reason.
    Keito
    Member
    1 year ago

    A little 3.2 update is up.

  • 3.2 Authors! Again.
    • I've moved code that is getting artists from e621 to the external file "ARTISTS.bat"
      • Now getting all the artists! For some reason, Adobe recognises n-1 artists (not less than 1 tho). Fuck Adobe!
    • "unknown_artist" is no longer an artist.
    • "avoid_posting" is no longer an artist.

    I've honestly tried to make Adobe recognise all the artists, but failed miserably.
    All the other software including Windows Explorer gets all the tags correctly. What a shame...

    The only improvement that comes to mind is a "hey bro there's updated\biggger\better version on e621 already!" notification, but i'll need some changes on the server side for that (they are planned already).
    If you have anything else to suggest, make it rain.

    Keito
    Member
    1 year ago

    I've added the updater component mentioned in previous post. To use it, simply run updater.bat

    UPDATE: It seems that updater might be very slow at the beginning. It depends on amount of files in the directory, and if it's taking more than one second to proceed each filename (1st stage), please tell me about that.
    If it's renaming files for more than 5 minutes, you need a fix.
    Temp fix: remove lines number 9-46

    UPDATE; Fixed in 4.1

  • 4 Updater
    • Updater is now included. You may think that it updates this tool, but it will download new pictures instead of the old ones saved locally.
      • Many pictures may have new, higher resolution, not watermarked, not fucked up versions uploaded to e621. Updater will find all the new pictures and download them, moving your old files to "backup" folder.
      • Still requires testing, since technically it will save parent of deleted picture. It may be the same picture, or it may be edited one (still i guess it should be better if the old one got removed from there)
      • Only works with md5 filenames at the moment.
    • Possible fix to authors: it should now get authors for not-md5 filenames too.
    Vayln
    Member
    1 year ago

    Great tool! Would you mind adding support to tag the date as well?

    Keito
    Member
    1 year ago

    To be honest, i have no idea how date works in JSON.
    I mean, as an example:
    "created_at":{"n":236133000,"s":1445010595,"json_class":"Time"}
    How does this translates into 2015-10-16 is a mystery.
    If i'll have some spare time to make this work, i will. Otherwise you'll have to live with year tags in "tags".
    I can copy year (only) from "tags" to the "date" if you want.


    Keito said:
    To be honest, i have no idea how date works in JSON.
    I mean, as an example:
    "created_at":{"n":236133000,"s":1445010595,"json_class":"Time"}
    How does this translates into 2015-10-16 is a mystery.
    If i'll have some spare time to make this work, i will. Otherwise you'll have to live with year tags in "tags".
    I can copy year (only) from "tags" to the "date" if you want.

    s is seconds since epoch (Jan 1, 1970 AFAIK), n is nanoseconds (so, this is the fractional part; usually can be ignored)

    That said, date doesn't work in JSON, there is no date type. Different applications simply use different hacks to store time in JSON, either as strings or as dictionaries. There are at least 3 common ways.


    savageorange said:
    That said, date doesn't work in JSON, there is no date type. Different applications simply use different hacks to store time in JSON, either as strings or as dictionaries. There are at least 3 common ways.

    Making a date object from a string is dead easy... wouldn't call it a hack.


    Lance_Armstrong said:
    Making a date object from a string is dead easy... wouldn't call it a hack.

    They're all hacks because there is no JSON date type and no 'one single obvious representation method'. Something like RFC 2822 or ISO 8601 is just less of a hack because it has some official weight.


    Keito said:
    To be honest, i have no idea how date works in JSON.
    I mean, as an example:
    "created_at":{"n":236133000,"s":1445010595,"json_class":"Time"}
    How does this translates into 2015-10-16 is a mystery.
    If i'll have some spare time to make this work, i will. Otherwise you'll have to live with year tags in "tags".
    I can copy year (only) from "tags" to the "date" if you want.

    created_at.s is what's called a Unix timestamp and is a simple integer representation of the time (number of seconds since January 1, 1970). Almost every programming language on earth has a function to convert it to any date format you need. Just Google "<language name> timestamp to date".

    Keito
    Member
    1 year ago

  • 4.1 Bugfixes
    • Removed filenames checks completely, since it works way too slow for me (and probably for someone\everyone else).
      • I don't expect everything to stop working because of that, but feel free to report any bugs.
    • Added some more comments to both updater and tagger.
    • No longer deletes old log after launching the tagger.
  • 4.2 Updater filenames
    • Updater will now calculate md5 if file is not named after the checksum and try to use it for search. (Same thing tagger does, now for updater)
    Keito
    Member
    1 year ago

    Tool is expected to be not working because of implemented CAPTCHA.
    I have not yet tried it out but it shouldn't work if e621\cloudflare is asking you to enter captcha.


    Keito said:
    Tool is expected to be not working because of implemented CAPTCHA.
    I have not yet tried it out but it shouldn't work if e621\cloudflare is asking you to enter captcha.

    I'm forwarding this to Dari, our sysadmin, to see if there's a way to fix this.

    Edit: The captchas should be less aggressive now. Please test it and let me know if it is working properly.

    Vayln
    Member
    1 year ago

    I'm getting an error 403 (Forbidden) when trying to use this script now.


    Vayln said:
    I'm getting an error 403 (Forbidden) when trying to use this script now.

    It's because the API has changes.

    Keito
    Member
    1 year ago

  • 5 GUI
    • GUI v1
    • Batch script is recoded into a proper EXE with GUI. It's slightly less miserable now!
      • Still not a virus.
    • Tagger and updater consolidated. Use the program and choose what you want to do.
    • xidel is removed, wget added instead.

    Screenshot

    Feel free to report any bugs and broken images! ;)

    Keito
    Member
    1 year ago

  • 5.1 Bugfixes
    • "Close app" question at the end of the proccess is fixed:
      • Window no longer duplicate if answered "No" after "Update" sequence
      • App will correctly restart if answered "No" after "Tag" sequence

    If you have any feature requests please post them here.
    I would also like to remind all of you that you should avoid sharing your tagged images.

    Keito
    Member
    1 year ago

    Bringing the heat again!

  • 6 GUI v2
    • WGET and it's libraries are removed.
      • Internet Explorer 3 or greater must be installed for this program to work. Firewalls or the presence of multiple network adapters may cause this program to fail. You can use WGET edition (see bottom of first post) if you have problems with that.
      • You can now use proxies! This program will use a proxy server to access the Internet if such a proxy has been configured in Microsoft Internet Explorer's settings.
      • 70-90% faster.
      • No longer outputs -user-agent=Keito/Tagtool1.0 for e621 admins. Sorry guys. Can't figure out how to restore this without WGET.
    • "Shutdown after" checkbox added. Will shutdown your PC when proccess finishes. Useful!
    • "Force remove old tags" checkbox is no longer checked by default. Should make the whole proccess faster. (1-2 seconds per image)
    • "Time elapsed" and "time left" counters added. Time left is very inaccurate as expected. They update after every image processed (not realtime).
    • ImagesList file is now correctly deleted when program closes.
    • Creating after_log.txt file after every launch. Might be useful for everyone who used "Shutdown after" and wants to know how proccess went after.
      • Only basic statistics for now. Time start, time end, files processed, files skipped.
    • Minor GUI changes.

    Screenshot

    I've also noticed that there are some invalid tags showing up, e.g. "\u00e9". These are incorrectly parsed special symbols. Pokémon and it's é is a good example of such tag.
    Please let me know if you encounter any and PM me with MD5 checksum or ID of that file to fix it. Thanks!

    Klaue
    Member
    1 year ago

    How did you get around the restriction they put in on only being able to search for a single MD5? just spamming the server with requests? I once made an offline image tagger (it's glorious days are over, nobody gave a shit about it and I was sick paying for the server for hosting it) and made a plugin for it to grab the tags from e621:
    http://i.imgur.com/36qdSf6.png
    but ever since they removed that feature and only allowed searches for single tags, it kinda just lays around on my hard drive, I'm too lazy to tag everything manually. If searching for multiple md5 tags works some other way, I'd probably use it again :)
    If you're just spamming server requests, then that's not working for me, a proper pause between requests would make it have ages and not putting in a pause would not sit well with me, I don't want to punish the poor server giving me the infos :)

    Also, out of curiosity, where do you put the tags? In the exif data? PNG does not have exif but metadata chunks, but gif have none, so can they not be tagged? or do you put it in an alternate data stream? I toyed with that idea, but it only works on NTFS file systems and all the tags are lost when you copy/move the images to any other, like to a USB stick. Or does it use an internal database so the tags are only visible when you open your program? (that's the way I went back then with my tool)

    EDIT: and why did e621 deem it neccessary to remove all spaces after my punctuation marks? lol
    EDIT2: and here they are again. what.

    Keito
    Member
    1 year ago

    Klaue said:
    How did you get around the restriction they put in on only being able to search for a single MD5? just spamming the server with requests?

    Yep, pretty much. Select file, get it's MD5, request JSON url (https://e926.net/post/show.json?md5=%filename%) using name without extension, if result is empty, request it again using MD5 i've got before, if it's empty again just skip the file. It's not really spamming since it takes 2 seconds or so for each file, but technically it is. :)
    I doubt it's hard for server to catch up with this tool working, i guess it's like 3-4 people browsing e621 at the same time...

    I'm not really sure about your problem with multiple tags, since i've never ever touched tags.json
    I get just enough info from show.json to make things work.

    Klaue said:
    Also, out of curiosity, where do you put the tags? In the exif data?

    Yes, it's actually mentioned in the OP post. I'll just quote it :)
    Using XMP dc Subject for tags and XMP dc Creator for artist. It is compatible with Adobe, Windows and Picasa. Probably with all the others.
    You are correct, i can't tag GIFs or SWFs. I have A LOT of pictures stored and GIFs are less than 1% of those so i can't really care about them. Plus, i don't really like GIFs as a format and I feel like they are obsolete.
    There's no database or alternate data streams. I'll add something like that if i'll be bored enough, but i'm happy with this tool as it is now (well except TODO features in OP post...)


    GIFs would be obsolete if there was a decent replacement.
    SWFs were supposed to replace them, but they're going nowhere fast, heh.
    WEBMs satisfy some of the same need, but not all.
    Same applies to APNG (covering basically all the use cases WEBM is bad at).
    Once we actually have reliable support for both APNG and WEBM across all major browsers, as well as in image viewers in the case of APNG, GIF will finally be obsolete.
    (then we need to work on replacing SWF with SVG.)

    @Klaue:
    That's a preview-only bug, ignore it.

    PNG exif: there are actually multiple ways to do exif in PNG. None of them are official, unfortunately.

    Klaue
    Member
    1 year ago

    Keito said:
    Yep, pretty much. Select file, get it's MD5, request JSON url (https://e926.net/post/show.json?md5=%filename%) using name without extension, if result is empty, request it again using MD5 i've got before, if it's empty again just skip the file. It's not really spamming since it takes 2 seconds or so for each file, but technically it is. :)
    I doubt it's hard for server to catch up with this tool working, i guess it's like 3-4 people browsing e621 at the same time...

    Oh, well, if it takes two seconds, then it doesn't matter too much. Question is why it takes so long though, jou only take the filename, right? even calculating the hash anew takes like 4 ms per image here.
    2 secs per file would mean it would take multiple days for my (kinda massive, granted, been collecting for years) local library

    with multiple tags I mean that something like that used to work before, now a year past:
    show.json?md5=%hash%,%hash2%,%hash3%
    (I don't remember how it was exactly, I just know I could check a hundred or so hashes with one server call)

    Yes, it's actually mentioned in the OP post. I'll just quote it :)

    Ah, my bad :) didn't read the readme, sorry

    For your TODO, I fear they're not really possible or rather, make not much sense. All the date e621 has is the upload date and, some times, if it is tagged, a tag for the year. I've seen years old pics uploaded and given a recent date. As for alternative sources, weasyl API has no way to get by hash. FN API is not out yet. FA has none. So you'd be out of luck there. Best bet would be to call a reverse image search, but TinEye wants $200 for even their lowest tier (500 searches) and google has no access to boot and likes to use captchas for sources it deems questionable. InadescentAPI, another one, wants 75 pounds for their lowest tier and claims to use, beside others, google, so my scam alert is tingling. I don't think you can really do that. At least not if the file comes from e621 and hasn't the ID of, say, FA in it ;)

    well thanks for the answers :)

    and thanks savageorange
    personally I'm a fan of APNG.. full color instead of just the 156/frame of gifs, true transparency instead of "transparent or not" and its files are smaller than gifs. Too bad the support for it is abysmal