Open Sourcing the Siboot Face Editor

Back in January I posted something on my website about a facial expression editor that I had built for Erasmatron back in 1997:

erasmatronface1 java-siboot-face_med_hr

I published a brief comment on Gamasutra announcing my willingness to lead a team to make this technology available to all. Basically, nobody was interested, so nothing came of it.

I ended up writing a little face editor in Java for use in Siboot. This tool allowed a storyworld builder to create, edit, and display faces. It didn’t produce photorealistic images: the intention was to get something more like “serious cartoon faces” that could exhibit a broad range of emotional expressions.

facemodel_med_hr

This face editing prototype is an ideal testbed for investigating the practicality of open sourcing the rest of the Storytron technology. It’s only about 1500 lines of Java code, but I cut a lot of corners in its development so a proper version would probably be larger by some order of magnitude.

I propose making this source code available with one caveat: I’m not going to simply place the existing code in a repository somewhere. It’s written in Java and the cruel truth is that Java is no longer a mainstream language (only “techies” can install Java on their machines). This would prevent the Face Editor from being useful to a large chunk of the public.

JavaScript seems to be a language with a better prospects – it’s ubiquitous, it’s fast and getting faster, and it’s a low-level as a web programming language goes. This leads me towards suggesting a project to convert the existing Java code to JavaScript.

But, as I wrote earlier, I will not do this myself. I want to see some commitment from the community. I want to know that I’m not just talking to myself when I write these posts.

Here my proposal:

  • We’ll discuss this thing for one week. All specifics are open for discussion.
  • On Monday, June 16th, we’ll end the discussion and I’ll assess the responses.
  • If a sufficient number of people offer their services to port the face editor from Java to JavaScript we’ll proceed.
  • If the discussion fails to generate enthusiasm I’ll set aside the issue of open sourcing the Storytron technology for a later date.

So let’s hear it: Do you think that we should use JavaScript? Do you think that the technology is too crummy to worry about? Are you willing to do some of the work to make this happen?

Face-1

Advertisements
This entry was posted in General. Bookmark the permalink.

38 Responses to Open Sourcing the Siboot Face Editor

  1. Bill Maya says:

    If I knew JavaScript better I’d say “Sign me up!” without hesitation. Three suggestions though.

    First, use something like Github for your source control (they have private as well as public repositories).

    Second, take a look at CoffeeScript, a language that attempts to mask some of “gnarliness” of JavaScript but compiles one-to-one into the equivalent JavaScript. I’ve heard good things about it and know some web developers use it to complement their JavaScript work.

    Third, read JavaScript: The Good Parts (for a basic introduction to the language check out Eloquent JavaScript).

    • furrykef says:

      I think LiveScript is better than CoffeeScript. I found a variable scoping issue with CoffeeScript that made me run for the hills. LiveScript is like CoffeeScript but not only fixes this issue but adds numerous other improvements to the language.

  2. furrykef says:

    The way 3D animators make faces is they sculpt their mesh into several different forms, each one representing a different extreme for one part of the face. They would have a face with raised eyebrows, a face with a big smile, a face with a big frown, etc. Then once all of these were made, they make each one into a blendshape. So if you want a character to smile just a little bit, you apply the big smile blendshape, but only (say) 25% of the way. That way it pulls the vertices 25% of the way toward a big smile. This same blendshape idea can also be used to specify different kinds of faces (as in the variable eye separation, mouth width, etc. in the Siboot face diagram).

    If that’s too complicated, there is a much simpler way, namely creating “bones” that manipulate the polygons. Raising an eyebrow is a simple matter of lifting the correct joint. This method could also be combined with the other method; for instance, using blendshapes to model the varying facial structure among actors, and using bones to model facial expressions.

    Either of these techniques could be used with 2D faces, because there’s no rule that says polygons have to make use of all three dimensions.

    Keep in mind that creating the face is the hard part, and it only has to be done once. For something like a generalized cartoon face, the user needn’t even do it themselves if we do it for them. After that they’re left with the easy part, which is pulling various features around and making faces with them.

    I think for this project to be worth the effort, it would have to represent a significant improvement over the techniques already available.

  3. Alex Vostrov says:

    I think that you’re 100% correct on the importance of faces, Chris. They pack an emotional punch that almost no image equals. A cheap, expressive 2D face tech is a very interesting prospect.

    Is It Useful?

    This is tricky. I’ve done a very cursory survey of face tech and most of it is 3D. In other words, you pretty much need to hire a 3D artist to make it work. This makes me very nervous as a developer, since 3D is a bottomless pit where money and effort can be poured endlessly.

    So, 2D is actually an advantage. Now, how flexible is the tech? It looks like an excellent fit for what little I’ve seen for Siboot images. My only concern is that it’s tied very strongly to the art style.

    Here’s a pointed question: why is this superior to me asking an artist to draw 10 expressions for every character? They’ll do a better job and I’m not certain that it will be that much slower for a skilled artist. The only benefit I can imagine is dynamic expression blending/animation (e.g. microexpressions).

    Porting Work

    Now let’s talk specifics, and I’ll be very straight with this. My personal interest is in making a great game with “real characters” (still deciding what the heck that means to me). The flip side is that I’ve little interest doing work for someone else’s project, unless it will be useful to me down the road.

    I’m certainly curious about your face tech and I want to play around with the Java face code. If it meets my design goals, I would port it to C++, which is what I use myself. HTML is a promising platform, but it’s still gestating and I have zero experience in working with it.

    That’s rather tepid as far as commitment goes, but it’s a realistic estimate of my motivations and time. Maybe there is some sort of happy medium, like Bill suggested. These days there are a whole bunch of intermediate languages that may be able to compile down to JavaScript or C++ or whatever.

    • crawfordchris says:

      You know, Alex, for Gossip I did exactly what you describe: I had an artist draw ten expressions for each of six actors. It was slow and expensive, but most important, it restrained my efforts. Once I had decided on those ten expressions, I was locked into them. Expansion just wasn’t in the works.
      This is especially important for communicating intensity of emotion. There’s a whole range of expressions between “slightly annoyed” and “furious” and you need all of them.
      Yes, there is a danger that any such system will be too closely tied to a particular art style. That’s one of the major issues that we’ll need to address in designing the general-purpose version of the technology.
      And yes, the whole point of this is to make something that people can adapt to different products.

  4. crawfordchris says:

    Now, class, let’s not see all the same hands every time…

    Kef, I very much want to avoid the principles you describe. Those principles are based on *physical* reality, not *emotional* reality. There are great facial images in which the eyebrows extend beyond the outline of the face. That’s a physical impossibility but it makes perfect emotional sense. That’s the kind of thing I’m driving at. I most definitely do NOT want this thing to be driven by any sense of physical realism. There are tons of face display systems that already do that, and there’s no point in trying to compete with these excellent products. What we need is something that a cartoonist would embrace, something that adheres to the principles of good graphic expression, not the principles of physics. Here are some examples of my meaning:


    http://josephpearman.blogspot.com/2013/05/cartoon-faces-week-5.html
    http://abduzeedo.com/comic-book-artist-skottie-young

    On the matter of languages, I have absolutely no religious or personal preferences; I am “equal-opportunity-ignorant”. I’ve already started looking into some, but one requirement is unbending: the system must be universal. That is, it must work on the great majority of personal computers. That wipes out a lot of languages and points us towards Javascript. However, I’ll surely look into those Javascript-variants you suggest.

    • Alex Vostrov says:

      I’m a bit confused by the “works on a majority of computers” requirement.

      That’s an excellent target to aim for as a developer and an artist – you get a larger audience! However, the target group for the face tech isn’t players, but developers. So I think the deciding factor is “where is the largest concentration of developers who might benefit from the face tech?”

      Maybe HTML-based games end up being the next big thing and Javascript is a great choice. On the other hand, I remember when Flash was big and nobody ever did anything important in Flash. Developers started their careers in Flash making small games, and then moved onto C++ or C#.

      I guess what I’m saying is that obscurity is more dangerous than obsolescence.

      • crawfordchris says:

        Hmm… Funny, I’ve always thought that it is obvious that the world is moving towards a web-centered environment. Back in the 90s, they were expecting that operating systems would be replaced by web-based systems. All you’d need was a machine with a basic web capability, and it would download whatever OS portions it needed as it needed them. Java was expected to be a major component of this. Microsoft took this threat so seriously that they went into all-out attack mode and did everything possible to stamp it out; I believe this is where the “Embrace, Enhance, Extinguish” slogan arose.
        Microsoft succeeded in deferring that, but it remains inevitable. (Microsoft has done more to retard the computer revolution than any entity on the planet.) Already we’re seeing lots of processing smeared out between the user device and the web, the best example being Siri. The app on the iPhone does only speech processing, sending speech codes back to Apple, where the heavy-duty AI resides. Is Siri an app or a service? The two blend together. Google is pushing hard in this direction, replacing on-board apps with on-web apps. This process will surely advance.
        It therefore seems obvious to me that the web is the future, and that pure PC-bound apps are slowly being eclipsed. That’s why I think that everything associated with interactive storytelling must be fundamentally web-based. Programs in the various dialects of C are dinosaurs because they’re dedicated to a particular platform and cannot operate with catholicity.
        I know full well that I am ahead of the times. Hell, I’ve been writing a book as a hyper-document for the last 20 years, and I refuse to consider publishing it on paper, because I know that the future belongs to hyper documents. As a consequence, nobody reads my book. Too bad for me. Too bad for them, too.
        We know that Java was the Great White Hope and it failed. Nowadays Javascript is the Great White Hope, and perhaps it too will fail. But one thing is certain: platform-dedicated software has no long-term future. None.

    • furrykef says:

      That’s certainly a fair argument. L.A. Noire touted its advanced facial technology, and I’m sure it was quite a marvel, but, for all the effort they put into it, I found the faces difficult to read.

      That said, I still can’t help but think there must be solutions that already solve the problem. I think the techniques I already mentioned would still work perfectly fine. If I wanted, I could probably make an abstract face that perfectly matches the diagram in the post and is perfectly capable of accommodating any cartoony expressions desired without regard for physical realism. Moving control points and moving bone joints (in the animation sense of ‘bone’) are two sides of the same coin, I think, except the latter is a system that is already supported by myriad tools, many of them freely available.

  5. crawfordchris says:

    Fellows, I looked at CoffeeScript and LiveScript, and was at first optimistic, but then encountered the DealBreaker: life expectancy. What happens to our work when (not if, *when*) support for these Javascript supersets become obsolete? I am very wary of depending upon somebody else to support something I’m using. I have been on BOTH sides of that fence and it’s a killer problem. I think it necessary to confine ourselves to languages that people will be able to find documentation for *at least* five years into the future.

    • furrykef says:

      Well, what does obsolescence entail? The inability to find a compiler and a reference manual? I strongly doubt that will happen. That would mean the github repository would have to be taken down, as well as all the mirrors of the repository and its releases. And there’s no reason for the github repo to disappear unless github itself goes belly-up, which is quite unlikely. And we could always maintain our own mirror in case something happens.

      So long as the source code exists, it would be possible to fix any bugs or other issues that arise. And the code would be trivially interoperable with any other JavaScript-based language, so one could gradually transition to another JS-based language if necessary.

      • crawfordchris says:

        Perhaps I misunderstand your suggestion. I looked at CoffeeScript and found a different language that is compiled down to conventional Javascript by a compiler. So if we write our code in CoffeeScript, and CoffeeScript disappears, then we’re screwed.
        Another objection, which I did not mention because I considered the first objection decisive, is that fewer programmers will be familiar with CoffeeScript (or LiveScript) than with Javascript. I have strong preferences here to stick with standards.

    • oneconch says:

      Besides kef’s points, it is also possible to access many copies of the content of nearly every website, LiveScript and CoffeeScipt’s documentation pages included, with often a dozen or more snapshots saved from every year. And considering the Internet Archive is a non-profit foundation with a $10 million operating budget and international backups, I don’t think they’re likely to go dark any time soon. As the saying goes, the internet never forgets. https://web.archive.org/web/*/http://livescript.net/

  6. MTL says:

    I’m of the opinion that the whole face issue is a rat hole. A great counter-example of this is Imgur. With one, count em’ one, good well presented face can move ppl’s hearts and minds really powerfully. The issue is not how do we generate faces, plenty of those (hell use meme generator faces/advice animals if you must), the issue is how do we use them. Having a good face editor is like have a good UI– a good thing for production, but not the point. Sure you want good thesbians, but you need great story first!

    As for languages, my vote is Javascript. For better or worse, Netscape gave it to us and it has stuck. Plus, now that it’s mature, it ain’t half bad. I dislike Coffeescript’s line-noisy-Forth implicitness (lexical vs dynamic scoping, with obscure syntax, really?). The issue isn’t whether its source goes away or not, the issue is how much stuff are you going to have to support in the future. More layers means more technology to break– witness Alex’s Chrome + Java taste of bitrot. I would rather be writing stories rather than toolchains and libraries.

    • crawfordchris says:

      Well, MTL, if you don’t think that faces are that important, then by all means refrain from volunteering for this project! Yes, of course we want good stories, and yes, of course, they must be our primary concern, but presenting the face is a critical part of good storytelling. Take any movie and count the fraction of frames devoted to faces. It’s huge.

  7. oneconch says:

    I’m certainly interested in helping out. I’m not a Javascript expert, but I have used it to limited extent in other projects, and I’m willing to learn. What I don’t know is Java, but it seems to exist something like a merger between C++ and C#…

  8. The first question this project begs is where you’re aiming to deploy. If you’re after web deployment (ie – the system running independently, client-side, in a web browser), JavaScript is (sadly) the only option. I say sadly because JavaScript has such a long history of security issues that many browsers disable it by default. Python is another option though, like Java, tends to be absent from non-geek systems.

    If you’re aiming for this to be used as an open-source system within independent software, I would argue for sticking with Java – no matter how much I personally loathe it. Java’s JVM operation makes it readily deployable on virtually every platform in existence and, unlike the wholly unrelated JavaScript, it’s structured enough to be easily portable to mainstream programming languages like C# and C++.

    A third option – which meets the longevity and end-user-ubiquity criteria – would be to use Flash. Makes things a bit more restrictive on the creative side of the equation, but I seriously doubt that Flash is going away anytime in the next decade or three.

    One last option to consider, if aiming for web deployment, is CSS3. CSS3’s 2D transforms are pretty powerful, lightweight, and cross-browser compatible. This would sadly still require JavaScript for client-side deployment, but also opens it up to most every server-side scripting language in existence – PHP, LUA, Perl, ASP, DNA, Python, Ruby, etc.

    Just something to think about…

  9. crawfordchris says:

    Thanks for the comments, Twyla. I feel strongly that we should design for web deployment, because over the long run, the web is the future. It’s almost a certainty that at some point in the future, there won’t be Windows or Mac OS, just Web OS. This was Sun’s vision twenty years ago and I think that it’s still on the mark. Of course, ‘someday’ could be a long ways off.
    Moreover, JavaScript seems to be establishing itself as something of a lingua franca among programmers: a language that most people know, but nobody necessarily agrees is the best.
    Hence I suspect that we’ll need to run with JavaScript.
    Your points re Java are solid, and I am not at all wedded to JavaScript; in fact, I am more comfortable with Java. It’s just that I fear that Java may have a shorter life expectancy than JavaScript. Does that seem right to you?
    Any other comments on Java versus JavaScript?

    • As much as I regard Java as the “red-headed step-child” of the programming languages, its JVM architecture has made it lingua franca for cross-platform development – relied upon heavily for mobile devices in particular. I honestly don’t see it going anywhere unless and until:
      a ) The entire OOP paradigm somehow drops off the face of the planet. and
      b ) The portable device market (smartphones, tablets, etc) evaporates.
      While both of these are potential possibilities, I honestly don’t see either occurring anytime within the next decade or two.

      I most certainly wouldn’t choose it for serious programming (I swear, sometimes programming in Java makes me feel like I’m trying to type while wearing boxing gloves) but, for applications such as this, Java is probably the strongest contender.

      Apart from security issues and the like, I suppose my primary gripe with JavaScript is that it’s just so damned sloppy! Compared to C/C++, C#. and Java, it’s so unconstrained as to encourage all the worst programming practices – far, FAR moreso than even the most free-form variants of BASIC from back in the day. If anything, I expect JavaScript to disappear far sooner than Java. As-is, a great deal of JavaScript’s functionality has already been superseded by CSS3 and XHTML5 – I doubt it will be too much longer until the rest follows.

    • Bill Maya says:

      I would vote JavaScript because of it’s ubiquity (and if you needed a server-side component there’s always node.js).

      But since those few of us who’d like to help out on the project don’t have JavaScript as our first language it looks like Java might be the choice if Chris wants to release this into the open source community.

      Chris, I vote for putting the Face Editor code out there in a public Github repository with instructions how to compile and run in Eclipse and see what happens. You gain some valuable experience with open source in an actual trial and it might take off. What have you got to loose?

  10. Chris Crawford says:

    It looks as if we’re converging on releasing this in Java in its current form. I’m ready to run with this, BUT in the opening post of this thread, I said that I’d wait until Monday to make the final decision, so I’m going to give people the next 72 hours to get in other points or recommend some changes.
    Here are a few lesser issues to consider:
    1. Messy/Clean. To what extent should I clean up the code, add comments, etc? Right now it’s in my tossed-together form and certainly needs some clean up. It’s teenager-bedroom messy, but can it be more like typical-bedroom messy or should it be more like barracks-room clean? Another way to express this question is, how many hours do you think 1500 lines of teenager-bedroom messy code should get in the way of cleaning?

    2. Generalization. This thing is written specifically for a single application. I’m sure that people will add abstractions that increase its utility, but to what extent should I do some of this?

    Just for fun, here’s the class that draws the face, in its current messy form. I scrunched the tabs down to 1 space to make it a bit more readable.

    There it is. Now the whole world knows that, all these years, I’ve been faking it as a programmer!

    • oneconch says:

      Hmm, that seems to have broken the formatting after the first public static code block. It’s fairly easy enough to navigate for a programmer nonetheless, although there seems to be a few too many closing braces at the end… I assume lines #428 and #340 should not be braces?

      • Bill Maya says:

        I may have done something to the source code when I moved it to a separate page. Will look into it

        Update: I fixed the formatting. Hopefully I got the indentation correct. Left braces at end.

    • Kef Schecter says:

      Well, with a repository on, say, github, there’s the option of putting it up as-is and allowing the community to clean it up with you, since you can “pull” others’ changes back into your own repository. github makes this sort of collaboration nearly trivial once (and this is a big “once”) you’ve gotten used to using git in general. It can get tricky if two people edit the same piece of code, though, since it can cause edit conflicts, so some coordination of effort might be necessary.

  11. Oh dear! very complex discussion here. I’ve been toying around with Chris editor for a while now and it’s very interesting and became very relevant when you understand Chris vision. I want to share some facts from my early mailing with Chris, just to wrap up some facts.
    – “I do know that most are 3D in nature and emphasize photorealism, which I reject.”
    – “I want something for ARTISTS, not programmers!”
    -“I want something that is optimized for the face”
    -“I want to build the fundamental components of facial expression into the software, but give the artist maximum freedom to create within the boundaries set by those components. ”
    -“I am not considering animation as part of my technology. This will be delivered over the Internet, and I don’t want to use all the bandwidth required for streaming video”
    -“not confine your expressions by anatomical considerations.”

    So, what I understand, the tool goals are:
    – Expression Design
    – For Artists
    – No Animation
    – Art style freedom
    – No anatomical considerations
    – Optimized for streaming
    – Web Technology

    This will let us digg deep on emotionally complex story design. Face expressions from characters will change live every time we make a decision. It will be dynamic.

    Hope this through some light to the discussion, and Chris I hope I explain it right!

    • Chris Crawford says:

      That’s an excellent summary of the goals of Face Editor, Alvaro. I hope that everybody pays close attention to those — I am highly averse to altering these basic principles. Does anybody see any problems with these principles?

      • Twyla Naythias Fox says:

        I’m a bit confused by the “No Animation” part. “Optimized for streaming” implies dynamic content, which would imply animation. If animation isn’t to be considered, it becomes static content which doesn’t involve streaming.

        If done right (similar to MPEG, where only the part of an image which changes need be conveyed), a simple update function is the only thing needed to include animation – potentially packaging a second’s worth of animation within the space needed to define the initial face.

        I could certainly see saving the animation component for a later date, though it seems to me that it should be a consideration for the initial development.

  12. Chris Crawford says:

    You’re absolutely right, Twyla. We’ll keep animation as an option for the future, but remove its prohibition from our current design principles.

  13. Well, after a week, we have the following responses to my call for people to participate:

    1. Bill Maya
    2. Alex Vostrov (slight commitment)
    3. oneconch (sorta)

    This doesn’t look promising. I have spent the last five days rewriting the code to make it more flexible and more understandable, but given the tepid response, I don’t think it’s worth the time it would cost me to manage the effort. I’m going to hold off for a while. Perhaps we’ll get some more interest in it at a later time, at which point I’ll re-open the question.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s