FFDB/code - design

This summary is not available. Please click here to view the post.

FFDB - progress

So, you might be wondering what's going on with the project, since my last post almost two weeks ago. Not all that much.

I've decided to think and write an actual document before implementing anything, but I do know the design will change during implementation, as my understanding of the problems and the tools gets better. I just didn't want to start coding before I was sure I knew what I wanted to code. So I have the introduction written. It's more Rationale than Requirements or Specifications (the Design part is not even started). It's a bit too formal for a personal post and a bit too personal for a formal document. I can't quite feel the reader.

Instead of doing my Passover homework (reading by all accounts an excellent book) I'm reading up on Web Services (beyond the buzzword).

I'm having a very difficult time deciding on the right technological approach: the API is in XML since I see my data best in XML form and the existing tools are indispensable in processing for presentation and achieving interoperability. But I don’t know of any good free (at least as beer if not speech) XML database, and anyway I think I can better tune a relational one. But of course, mapping XML into a relational database might be bad for both performance and maintainability. Also, I think Python is cooler than Java, but the benchmarks definitely favor the statically typed language. And of course, PHP is the popular choice, but it always felt like a giant kludge to me, and its latest attempts to imitate Java only serve to point me to the real thing.

One thing I can tell you right away: dotNet is not even considered, because I’m biased.

FFDB - An experiment

If I am to design the FFDB, I will need your input. Lots of it.

Ideally, the design should be done openly on a wiki, where everyone can say whatever they like about the goals, methods and parentage of the team members.

But, I can't afford decent hosting for a wiki at present and the development team consists of only one member, so we'll do without one.

I will post thoughts, ideas and samples here and you will reply with criticism.

The thing is basically about fics. What should it know about a fic?

1. author(s) – are cooperations separate authors or should a story have a list of authors?
2. title – utf-8 string
3. is it complete? – boolean
4. number of published chapters – numeric. if = 1 and fic is complete, special case of cookie/ficlet, no table of content required.
5. expected/required reader maturity rating – one of: G, PG, PG-13, R, NC-17, X
6. explicit warnings – for objectionable and or unsafe (work/school-wise) stuff like: slash (separate for femme-slash?), underage sex (what's the age of consent in Potterland? Leaving Hogwarts?), mild fetishism, violence (BDSM? Rape?), bestiality, incest.
   These can be anything from verygraphic to barely mentioned or even implied. Should there be a rating/scale?
7. genre – <list of genres>
8. spoilers for books <list of books>
9. main characters <list of character> – should original characters exists separately or be grouped together under the label OFC (OMC)?
10. ships <list of ships, i.e. character groups> – should there be a list of ships?
11. word count – numeric. SUM(word count of each chapter)
12. first publish date – date. MIN(publish date of each chapter)
13. last update date – date. MAX(publish date of each chapter)
14. canon-compliant up to which book? – book or numeric?. this can differ between chapters, how to treat it for the entire fic? (max, min, average)
15. language quality – this can differ between chapters, how to treat it for the entire fic? (max, min, average?)
16. edited by <list of authors, personas or real people or combination thereof?> – this can differ between chapters, how to treat it for the entire fic? (union on lists?)
17. is it part of series? – a series and item number
18. was it written for a contest/challenge of some kind? If so, which place did it take? – challenge/contest id, possibly numeric place
19. archived at:
    here, should it just be a list of URLs or do I handle major archive sites in special ways? Like:
    1. SQ fic id
    2. FA-member URL
    3. Y!G name
    4. FFN id
    5. Checkmated id
    6. Lots of other sites ids or URLs
    This can be a list of dual thingiescontaining archive site id and specific fic id, handled by some business logicwith a reg-exp specific for each archive-site.
20. Other stuff I forgot

About people:

I think there should be a separation between fandom personas and IRL persons. We are all slightly different people when online, even those of us who do not actively re-invent themselves. Fandom personas have fame, LJs,f-lists, fangirls, mod duties, published works, reviews. IRL persons have names, ages, genders, home addresses (a not necessarily complete collection of: continent, country, province, town, street, home), private emails, ICQ UINs, AIM, MSN and Y!M screen names, social software ids (Orkut anyone?), marital statuses, children, birthdays, and social security numbers (kidding!).

What data should there be stored, and what should be available to mods and the general public, about each? Should the connections between persons and personas (any better terms, please?) be straightforward 1:1 or is it possible that multiple persons make up a persona (collectively working?), or that some person has virtual split-personality disorder? Can/Should the FFDB be aware of such cases?

How to see if it's right:

Think about search. Imagine what you would want to search for, any whether this data model has a chance to find it for you, if such a thing exists and the entire fandom was indeed included in the repository.

For example, a query:

Find emails of persons (for flaming purposes, probably) who have personas who are authors of fics that are: Rated NC-17, include bestiality AND incest AND underage sex, of Plot What Plot genre, contain no spoilers, whose main characters include Ron Weasley, Ginny Weasley and Giant Squid, ships include Ron/Ginny, word count is over 50K, have apparently not been proofread ever (probably because the content is too arousing to read over-and-over again), are completely AU, and archived exclusively at the Quill.

We all hope this always returns “none found”, but I wouldn't include it in the test cases. ;)

I know there is no way you have understood everything, because I have not tryed to explain. But I want you to get involved, so you are hereby asked to ask questions. By your comments I'll see if there is any interest. If so, we may have some progress with this thing. Just watching/reading will not get you anything, because there'll be nothing to watch - I won't do any thing. I'm really good at it.

RJA emailed me and I was able to help her a bit with the archives template (after struggling with my highly irregular archive template for this blog, I feel I'm fairly knowledgeable in this). It feels so good knowing I've learned something for being able to do stuff and help people and not just for the bragging rights.

BTW, I feel like I should review the new SQ v3 1.0.1 engine, but there's little I have to say. It seems very nice, though I would've changed some stuff layout-wise. Forced me to reconsider some of my own work.

Am still pondering on the idea of compressed chapter transfer, but it looks like the viewer must be made in Java for such a thing to work, which makes the page (applet, rather) too "heavy" for my taste. But the idea of a JS php-generated navigation system is getting along nicely. Makes for fast refreshing and lower server load (also conserves bandwidth) so I think I'll use it.

Would've liked to host actual content myself, in parsed form and possibly even compressed (just look at what solid rar3 can do to html files!), but it appears I will never posses the storage and bandwidth capacity. I wonder who pays SQ's and FA's bills. I know RL finances are not a popular topic in the fandom, but the bottom line is: someone has to pay real money for the sandbox. A&Z don't sound like two girls who can spend a fortune on hosting fees...

Am considering totally abolishing first person pronouns, and not for bandwidth considerations. Also am beginning to seriously consider a way to have both singular and plural second person pronouns like in all normal languages. Showing respect by using plural instead of singular is one thing, but not being able to make the distinction is quite another. Though, I really don't fancy using y'all...

Am going to further restrict anti-spam filter on the hotmail account, so please use the other address if you don't want to wait while I fetch your mail out of the bulk mail folder.

Have considered using PGP, but have decided is pointless since most correspondences are not geeky enough to bother with it.

Edit:Apparently I wanted to design the perfect fanfic archive site to pitch it to various folks (for free). I have developed javascript that generated nice TOC pages and navigation links in a layer that hovers at the top of the view port. It used anchor names of URLs to store data in bookmarks and was supposed to be generated using php from a database, though the backend was never developed. The realization that I don't want to store the fic itself but only the metadata evolved into the idea of crawling fic metadata from various archives into a single database to search for fics fandom-wide.