NekoSnowy said:
If a suggestion would be okay, would it not be possible to take advantage of NTFS's support for storing property data in an annex to any file, thereby enabling proper metadata for the formats not currently taggable, mainly gifs, a real pain tagging all these gifs one at a time.

Windows doesn't normally allow for this except for a few file formats because the feature was pretty much only 99% completed, but there's a free program that finishes the feature thereby allowing all files to have metadata on them that can be searched through windows search and explorer like such~ http://i.imgur.com/HBzI9Wn.png

This sounds like basically the same idea as Ext4 xattrs. The principle problem with storing important data in xattrs is revision and replacement. For example, you load an image in, edit something, and save. There are two possible ways a program would implement that:

  • Truncate the file and rewrite it from the start
  • Write a new file, and when it's finished writing, remove the old file. Finally, move the new file to the same location as the old file.

The former approach would preserve xattrs, but it is not safe (an error during writing could result in a completely broken file). The latter approach is safe, and thus by far the most common approach used. But it only copies xattrs if the program explicitly transfers them over too. I would be very surprised if this is any different for NTFS properties.

The other main concern being if you zip/rar/7z them up, are the properties preserved? Again, this depends on the individual archiver to 'do the right thing', otherwise you just won't get this data on the extracted files. If you change filesystems, you may not get the data even if the utility did store it. Or you might get just part of the data, as different filesystems have different restrictions on attribute size.

All of the above is by way of saying you need to be very cautious about storing data in anything that isn't strictly an ordinary file.

Extended attributes / properties are, AFAIK, in practice mainly used for the purpose of caching expensive calculations. Search tools extracting key information from files is the classic example of this.

EDIT: OP, you might want to look at this page , which basically takes the premise 'We want tagging via extended attributes.' and really looks into what is involved. Then it backtracks a bit and looks at the question of "what can we currently get?". You might not want to read -all- of it, but it gives some solid context.


It is a concern when programs, rather than update the file in place, save updates using a write new file, delete original file, rename new to original method. The former would leave the metadata in alternate data streams intact as far as I'm aware, while the latter wipes it out.

For archiving, you have to store the files in a WIM with the store NTFS alternate data streams enabled, and then compress that, which will preserve the metadata that's inside the NTFS ADS. As a backup, you can have filemeta, the program I mentioned earlier, export the metadata from the NTFS ADS to an XML file, and then filemeta can import it back to the file.


Honestly none of those looks good for me. Most of them are tied to using some kind of database or attributes and 3rd party software that may or may not work for some people. Supporting that kind of stuff looks painful.
I guess i'll put alternative tagging methods to the end of todo for now... Maybe i'll find some decent method later.


Keito said:
Honestly none of those looks good for me. Most of them are tied to using some kind of database or attributes and 3rd party software that may or may not work for some people. Supporting that kind of stuff looks painful.
I guess i'll put alternative tagging methods to the end of todo for now... Maybe i'll find some decent method later.

Yeah, it's pretty painful, and nothing is great. You could maybe try making something that just downloads the tags into XML files that filemeta can recognize so it can import the tags to the files for you, I don't know, that's all I have. I wish you luck in figuring this out, not an easy task~


I'm not familiar with how to view tags or how to take advantage of using them. Are the tags only available through certain programs? Do they have to be enabled? It doesn't appear like they can be searchable but I don't actually know. What are some of your tagging workflows? What do you use them for?


Artifice781 said:
I'm not familiar with how to view tags or how to take advantage of using them. Are the tags only available through certain programs?

Mostly, but systems like TMSU that employ VFS enable you to eg. search by tags via browsing the filesystem.

Do they have to be enabled?

I have no idea what that would mean, unless you mean in the context of the options of a particular program.

It doesn't appear like they can be searchable but I don't actually know.

Yeah, they are searchable. Would be no point otherwise IMO. Although the embedded-tags approach used by e621 updater can be a bit slow to search (data must be extracted from each image in order to determine whether it matches the search)

What are some of your tagging workflows? What do you use them for?

For images (downloaded images, my art), I have it set up so that the tags of the currently viewed image is displayed in sxiv's status bar. When I type an alphabetic character, it opens an interactive tagger (add tags by typing, with autocompletion etc).
If I want to find files by tag, I either mount the TMSU VFS, eg tmsu mount /tmp/tfs and browse that, or just do the search on the commandline, eg tmsu files apple pear not banana.

In my file browser (SpaceFM), I have it set up so that Shift+Space triggers the same interactive tagger on the active/selected files. I also have a script bound to Shift+Keypad_0 which simply displays the tags of the selected files in a basic UI.

Overall I really use it for.. any and all tracking. Since I can in effect associate arbitrary text data with any file, it's easy to use this to:

  • rate quality of things
  • Track admin work (things that need organization, deletion, etc)
  • Track reading progress (eg. via a tag+value pair: 'p=50')
  • Add crossreferences to other files by referencing their fingerprint hash
  • Arbitrarily group project files without needing to have them all in the exact same directory.
  • yeah.. any and all bookkeeping that needs to be searchable, basically.

If you are asking specifically about this app, then...

Artifice781 said:
Are the tags only available through certain programs?

They are available through the most of file managers and image viewers. For Windows i use explorer and Picasa or Adobe Lightroom. For Android i use F-Stop. They should be visible in most of the programs you may use for viewing images tho.

Artifice781 said:
Do they have to be enabled?

Nope, if your file\image manager supports tags (XMP in particular), they should be visible. Well maybe they are hidden somewhere in your software's interface.
In Windows Explorer for example you can enable "Tags" column in details layout. That's not very useful but you can see that your images are tagged at least.
To make use of them there, see the next quote.

Artifice781 said:
What are some of your tagging workflows? What do you use them for?

The easy way (i use in W10): just open the folder with (tagged) images, go to search panel → other properties → tags. It basically just adds "tags:" to the search field. Then after that i type, for example, "female domination". Then after a minute or two i can launch a slideshow and enjoy some femdom action, lol.
For Adobe Lightroom it's easier: there's a list of ALL tags from imported folder, so i just click checkboxes near tags i'm interested in and others disappear. Then i can launch the slideshow.


Quick tip: If exiftool isn't writing tags to your files, make sure your working path/directory doesn't have any spaces in it. No folders with spaces. "D:\this folder won't work\0C19A2AA4533C71961ED6E468E0A42C4.jpg" won't be tagged.

[Edit] I got jpgs to work, but I can't seem to get exiftool to tag png format. This seems to be a known issue.


Artifice781 said:
Quick tip: If exiftool isn't writing tags to your files, make sure your working path/directory doesn't have any spaces in it. No folders with spaces. "D:\this folder won't work\0C19A2AA4533C71961ED6E468E0A42C4.jpg" won't be tagged.

This is a quoting bug, and almost certainly e6updater's responsibility. If you are using exiftool independently on the command-line, you'll need to doublequote such paths (as you did in your post)

[Edit] I got jpgs to work, but I can't seem to get exiftool to tag png format. This seems to be a known issue.

Works OK for me. AFAIK the bug is in Explorer (or more properly, the bug is in the PNG spec not standardizing this stuff..)


Artifice781 said:
[Edit] I got jpgs to work, but I can't seem to get exiftool to tag png format. This seems to be a known issue.

PNG's should work if JPGs work for you. But they are not displayed in many programs. See post #200333
Try file properties in photoshop.


downloading a program from an image booru without source code published is begging for something bad happening to your machine - virus scans or otherwise.


fewrahuxo said:
downloading a program from an image booru without source code published is begging for something bad happening to your machine - virus scans or otherwise.

Yeah, whatever. Not going to bother with other files in package. Feel free to scan them. :)
Or you can just manually do what this program does - i don't mind.


Keito said:
Yeah, whatever. Not going to bother with other files in package. Feel free to scan them. :)
Or you can just manually do what this program does - i don't mind.

there are things that a virus scanner won't pick up - such as accessing your files, getting metadata about your computer, and sending information about such things - which won't be detected, and never will be due to the source not being released.

even if a program doesn't do these things, the idea that a program has the capability to is cause for concern.

all it takes is a GitHub profile with the source released to make a trustworthy program. surely you've heard of GitHub?


fewrahuxo said:
even if a program doesn't do these things, the idea that a program has the capability to is cause for concern.

I'm pretty sure that most of the people who downloaded this don't have these concerns. You can also check if this program connects to anything that's not e621.net: it's not exactly hard to do. If you find any addresses that are not affiliated with e621, i'll release the sources right away. :)

fewrahuxo said:
all it takes is a GitHub profile with the source released to make a trustworthy program. surely you've heard of GitHub?

I've heard about github but i don't want to bother with uploading anything as long as i keep updating this program.
First of all, my code is so shitty even i blush when i look at it. I'm not prepared for 10000 optimizations it needs. I also don't want anyone to fix them for me because it will be hard to realize what was changed and why - i won't be able to add functionality because i won't know what's where and what does it do.
As long as it's my hobby and the first proper GUI program i've coded i'll prefer not to share the sources. But yeah, if i abandon developing it, i'll release the sources: it's mentioned in the OP post.

(btw it's quite easy to get the source code from exe: it's an AHK script packed into EXE with AHK's built-in tool and compressed with MPRESS. If you really want the code, it won't take more than 10 minutes to get it)

Artifice781 said:
"D:\this folder won't work\0C19A2AA4533C71961ED6E468E0A42C4.jpg" won't be tagged.

I see the problem in my code, but i'm not sure how to fix it yet. :)
I'll look into it next time i update something.


Just found this tool that does exactly what I was looking for: uploading saved images to favourites. Nice job!

Just one suggestion mentioned earlier in this thread but not addressed - logging of files that aren't found on e621. It'd be really helpful to know which few files out of hundreds haven't been favourited.

Keito
Member
5 days ago

Dirtpaw said:
It'd be really helpful to know which few files out of hundreds haven't been favourited.

It's actually not that easy to do because AFAIK server returns nothing after POST request. I'd have to check if amount of favourited pictures has changed after each request or batch check them after process finishes... I'll think about it. :)


I implemented something like that as part of my system.
Basically, it does the fav:USERNAME query, fetching all pages, and then extracting just the URIs. It then reads a list of md5sums corresponding to on-disk files (this is cached, and updated as needed by digup )
It iterates over the list of URIs; for each one it extracts the md5sum. The URI in question is output if its md5sum is not in the list of on-disk files.

So what I implemented is basically the opposite procedure, but is simple to invert. Hope that helps give you some ideas.