Nexus One running Flash Player 10.1 from Ralph Hauwert on Vimeo.

This is a video of the Nexus One with a pre-release version of Adobe Flash Player 10.1. It does not represent the final quality of the Flash Player on this specific device. But it's pretty good as is!

This is the Nexus One running 2 demos. First a Nexus one specific version of the "Milkyball", which is adjusted in resolution and screensize.

Second a non-adjusted version of FlashModPlug, an experimental project I'm currently working on. It's a port and wrapper of LibModPlug, running inside the Flash Player through Adobe Alchemy.

Conclusion 10.1 on the Nexus is really, really fast!

(go to vimeo for the full resolution version). (link to nexus demo)

Related posts:

1. Triangle3D : Small Flash 10 3D demo. Over the last days I've spent some time twittering...
2. Goodbye, Papervision. Last week at Flash on the Beach I had...
3. Help me test! Flash VSync and Screen Tearing. So, here's a post where I am asking you,...

Long time no see! I've been busy, sorry. First off all, I want to congratulate you on the great show today. Wow, that iPad is truly one of those things everyone is going to want to have, like your previous successes with the iPod and iPhone. I take my hat off for that! And wow, what a lead up to the show. Probably the first hype of 2010. And do you guys know how to keep a secret and then give a great show unveiling it.

That being said, I felt as an avid Apple products user as well as your friend, (you know I own a total of 3 Macbooks, a plethora of Airports and have converted nearly my entire family to buy one of those nice machines of yours) (actually it was Grant who got me into buying one) it's my duty to inform you that I think something went wrong during your presentation. When you were showing the web capabilities of the iPad, something was missing in it's browser (see screen-shot above). As your keynote and product presentations are normally flawless (ah, well, maybe not always), I think you might have missed this one.

You touted the iPad's (great product name, btw) web capabilities as being amazing, perfect, you know, the regular Apple thing. But during the presentation I couldn't help but notice that little "missing plugin" logo we all know from the iPhone. Now, I was thinking that this might have been one of those very exotic plugins of back in the day, like Director or Realplayer. But, as it turned out, the missing plugin was the Flash Player.

As you've started the introduction to the iPad with a reference to Netbooks, and we can safely say that most netbooks support Flash, I think we are both safe to assume that to be on par with those netbooks, the minimum you should have is a fully featured browser?
I mean, this is not some kind of small screen device with limited capabilities in terms of performance and graphics. It's a fully featured machine ? Be it with that amazing iPhone touch interface. And be it in a slightly new package intended for true casual and business use. (Honestly, you kind of confused me, games and reading books ? I thought the gaming generation didn't read and vice-versa, ah we'll talk about that after you write me back on this letter).

Anyway, I think it must have been a demonstration error, because it seemed like the iPad didn't support Flash?

Mythhackers.

Ok, now, who am I kidding. I know you're a perfectionist. And you're a man of subtle remarks too. Between you and me, you can admit it. We both know that little missing plugin logo wasn't an accident. I think we both know why you did it. It must have been your kind of humor, getting back at that little prank  Kevin and Johnny of Adobe pulled on you at Adobe MAX. (see picture above too). You have to admit, that was funny. Man, we all laughed about that one. But hey, I always think that if they're teasing you, they must want your attention. All in all, they didn't mean any harm.

(I do agree with you that they should've spent some more cash on the explosion effect in the video, being the maker of some of the most premiere video effect software out there, but hey, it was a good prank video, executed very well).

Now, given some of the easter eggs in Mac OS X, which essentially is your baby, through that other thing you did, with that next-cube thing you were so enthousiastic about back in the day, I take it you have a fair amount of subtle humor in you. It always reflects in your letters and I always have to laugh when I think about how you made that joke about a mouse only needing one button. But, as always, there's a time for fun and a time for seriousness. And you seriously can't mean that the full featured web browser doesn't support the biggest plugin out there?

If you do mean it, I can understand why. I've never really bought that, "Flash is too slow for the iPhone" story you told everyone at that party. I also know for a fact that RIM, Nokia, Samsung, NVidia and all those guys also didn't believe it.

And yeah, I know that little stunt Adobe pulled with compiling directly from Flash to iPhone isn't the same as running it in the browser, but hey, it is the output of Flash running on an iPhone. And surely there's a lot of content which can now be readily available in the App store. And if that puny little ARM processor in an iPhone can, then surely that incredible piece of custom silicon you so proudly called the Apple A4 can. (it's ARM too, right?).

Your nightmare ?

Ahh, going off track, I was saying that I understood why you would not want Flash. I know having rich, web enabled apps which don't go through the App store can be a scary thought, food for nightmares. I know you've had a thing for nightmares, especially after Monster's inc. But Steve, I have to tell you. A device which is intended for casual and easy use with all the slickness Apple always brings, but doesn't support so much web content? Come on, what will the people think? Those netbooks you called slow and PC-ish, can run Flash, but Apple's state of the art technology can't?

Yeah, I know it's hard to have that piece of proprietary software running so much content essentially being a separate platform, while having no control over it. But that's why Kevin and Johhny where throwing those jokes at you. They want to help you, really. And again between you and me, isn't it true that you're always selling computers when those guys at Adobe come up with a new version of their creativity software?

I would like to also come with a more serious note, as your friend, I know I can remind you of this one. Do you remember that when I was still using a PC, we were always talking sh*t about those guys at Microsoft for dominating the market? About how they pulled that monopoly thing on Netscape with that "lovely" (I still remember the word you had for it) browser we all still despise, Internet Explorer. And what they did to Real by bundling that Mediaplayer? Well, as it turns out, they kind of ran into some anti-trust issues with the European Union. I hear it was quite costly for Bill. While I agree the iPhone was a completely different thing, you can't bring out a computer and expect everyone to think it's just a big iPhone and then pull all that stuff off that our friend at Microsoft once did, right?

Yeah, yeah, I know what you're going to say. Flash is proprietary too and there should be an open alternative to it. But honestly, in terms of open initiatives competing with it, it seems those HTML5 guys are still not ready for it, and although having H.264 video support native to the browser is nice, it hardly replaces what Flash can do right now. Not to mention all that content already being offered by Flash, people can't see on the magnificent iPad. If pure HTML5 content is even 1% of that, we both know that would be a lot. I do feel you should be open to any plugin, honestly, so the guys at MS/Silverlight and Unity3D, for instance, also get a shot. All in all, it's your product's users who are going to suffer for it otherwise. Now, as I've made my living building Flash apps, of course I am biased, but doesn't every developer deserve a fair shot?

You're the man!

You were always this idealist, but this thing is making you look like you are just plain greedy, at the cost of your end users, not being able to access all web content. Give me back the idealist Steve, I love that guy! (I've included an old picture of you to remind you of that man).

Now, you know I do admire that you're not like all those others and you're doing things your way (and it seems to work), I think you're going down the wrong path on this one, or this might have been just an error. If it was, just fix it before release, and we'll keep it between you and me.

Sincerest Regards, your friend,
Ralph.

(quick note, can you make Johnathan stop saying the word Magical, every time he's talking about one of the products he designs ? Frankly, it's a bit annoying. And he's not designing unicorns or leprechauns.)

(quick note 2 : if you do manage to talk to Kevin or Shantanu at the next party, ask them if their engineering people can talk to your engineering people about that video hardware support and performance issues, I think it would make many of our friends happy).

No related posts.

Milkyball, a further extension of my Triangle3D experimentation, used in my presentation of end 2009. Click on the shape to see the magic vertexmap.

PixelBender + Flash raytracer. Work in progress.

I posted both of these earlier on my twitter (@unitzeroone), but they didn't make the blog as of yet. More details in future posts.

Related posts:

1. Triangle3D : Small Flash 10 3D demo. Over the last days I've spent some time twittering...

(people who want to know more would refresh after seeing the pieces in full)

No related posts.

So, here's a post where I am asking you, the Flash community, for a hand. Any of it is truly appreciated and will hopefully get some attention and maybe even a solution to a problem which has been in around in Flash since as long as I can remember; screen tearing.

First, what does tearing mean in this context ? Read this wikipedia article. In this case, it's a visual split along a vertical line in a Flash movie. It's been a known issue with Flash since quite some time (always as far as I know), with improvements over several Flash player versions, but I haven't had to deal with it much lately....until now. I'm in a project where this visual artefact is a major distraction from the content and the experience.

All in all, this seems to be the cause (at least, my analysis of it) of tearing in this case; Flash internally rasterizes a screen, and pushes it to the screen buffer. Internally, this should be synced to your display refresh (60hz or 72hz, for instance), so that the buffer doesn't get updated halfway through the videocard building up the frame. But on some browsers and platforms it does update during the video card frame refresh, resulting in two rendered frames being displayed at once (or better; 1 frame being updated to the next while drawing to screen), with an optically visible vertical split visible between the two frames. This problem has been around for a long time. It's more prone to show up in content with "tall" items in it, fast frame rates with large portions of the screen moving, and during horizontal scrolling of displayobjects.

The picture above shows the problem in action. It's a picture taken using a regular camera pointed at the screen, as this problem doesn't show up in screencapture software. It shows two different frames of the testing movie; both a black and white square, updated halfway through the screen. While the position of where the slicing happens changes, in many single frames, 2 frames will be displayed at the same time.

I'm seeing problems with the latest 10 Player on Windows XP (Firefox), Mac OS X (Firefox, safari doesn't but "stutters), and Windows 7. All in all, this is a graphical issue which has been around for a long time and should be solved. It's definitely a Flash Player bug and with FlashPlayer 10.1 around the corner, I would really love to see this fixed. As well as being hopeful for finding a solution for current player/browser configurations. But testing results are varying and that's where I am asking you for help, by running 2 flash movies and posting the results to my comments.

Over the last days I've been frantically testing several configurations and options, I would hope force vsync with the flash screen buffer, amongst which the following :

• Adding a video at 30fps to the stage (no result)
• Adding system text to the display list (seems to work on windows xp)
• Several configurations of stage.invalidate() and Event.RENDER (no result)
• Different drawing / caching methods.
• Wmode="direct" for flash player embed options. (result across the board!)

The last option is the only thing which seems to work on all testing configurations, but comes at a cost. Switching to wmode="direct" seems to force vsyncing and this would adhere to what has been said about direct wmode:

"This mode tries to use the fastest path to screen, or direct path if you will. In most cases it will ignore whatever the browser would want to do to have things like overlapping HTML menus or such work. A typical use case for this mode is video playback. On Windows this mode is using DirectDraw or Direct3D on Vista, on OSX and Linux we are using OpenGL. Fidelity should not be affected when you use this mode." [Link to Tinics blog]

"Hardware offscreen buffer
Software renderer, perfect fidelity" [link to Jim Corbett's presentation (ppt)]

As opposed to the wmode="gpu" flag, which moves rasterization to the videocard (at a hight cost in most content cases, also altering the appearance of your content depending on the videocard), this option still uses the Flash Software Rasterizer to build up content for the screen. But on both my Mac and PC testing this comes at a performance cost of several frames per second, up to about 15fps on larger screens (1280x768). Also, I'm not sure about the results of this on machines with no gpu. (mostly, cheap on board graphic chipsets, which share system memory). While most of the time, your content type might vary in terms of results with either GPU enabled wmodes, in this test that shouldn't be an issue.

Obviously, abusing wmode="direct" to force vsyncing on the Flash Player isn't ideal, but it seems to work. In an effort to survey the results of this, I've built a very simple test swf. What I'd like you to do is run both swfs and copy the text field of both and paste them in my comments. Although I'm already in contact with Tinic about this, there's currently no JIRA bug describing the problem. With the results you have, I'd like to put up an official bug there, and use your results to describe what systems are impacted.

Firefox seems to be the main culprit. Since FF 3 there have been a number of big issues with Flash and Firefox; for instance, in some cases a right click on the flash player will cause the updated areas in the Flash movie to render as an opaque rectangle with the background color. In another case, Flash doesn't respond to rollover events and doesn't even render without a mouse click occuring.

Please help me and hopefully Adobe identify and fix this problem, by running these tests. Here's what you should do :

• Run both movies
• Look for tearing. Typically, in these test, instead of steady flickering, you'll see a moving vertical tear between the black and white area. It is essentiall you identify tearing correctly, so make sure you know what it looks like (picture above is a still). The effect of tearing shows even better on my mac, when you move the firefox window so that the flash movie intersects your screenborder, halfway splitting vertically through the flash movie).
• Copy and paste the textfield contents when the test is done, on both tests to my comments. Fill in the required 2 fields, and if you can, what type of graphics hardware your machine has.
• Send this link to other people who are willing to test (and be part of the solution, making Flash a better place!).

Test with wmode opaque

Test with wmode direct

[WARNING : if you are prone to visually induced epileptic fits, this might induce one. Don't run it then]

(don't forget to post to the comments).

Related posts:

1. Triangle3D : Small Flash 10 3D demo. Over the last days I've spent some time twittering...

Over the last days I've spent some time twittering back and forth with Eugene. His supershape demo is very cool and we spent some time 'tweeting' about shaders. This lead me to upload and twitter one of my experiments which I used as part of my FOTB / MAX presentation. Judging by the huge number of retweets it seems people enjoyed it a lot, so I figured that it was worth a blog post. I started working on this experiment a while back after toying around with OpenGL, OpenFrameworks and audio reactive shapes and that inspired me to do something similar in Flash. This is an unfinished demo as it lead me to pursue something else, at which point I stopped working on it.

In the session where this came from, specifically this part, I shared information about the path of yet another demo, where this experiment led me too. Some of you who attended Flash on the Beach, might recall it from the Adobe keynote, where it was aptly named "milky ball".  I'll present on it one more time on the 3rd of December at the AUG XL event, in Amsterdam. I'll post more of the session's demos with explanation, when I have the time.

I do want to clarify that the code base I've created for this, which I currently call Triangle3D, is not to be a new Papervision3D, Away3D, Alternativa3D, Infinity3D, Sandy3D, Null3D or Yogurt3D (I'm probably missing a couple, sorry, have your way in the comments). It's my experimental playground of classes to play around, with no care for architecture restrictions or a generalized API. It's all about me playing and enjoying. Pure experimentation of what can be achieved with Flash 3D and hoping to find something beautiful in it.

Orignal unfinished demo

Demo with adjusted lightmaps from my new mirrorball.

(here's the shot of the mirrorball, on flickr).

The music track is Sixtyten by Boards of Canada.

Related posts:

1. Flash Player 10.1 pre-release on Nexus One Nexus One running Flash Player 10.1 from Ralph Hauwert...
2. Goodbye, Papervision. Last week at Flash on the Beach I had...

My favorite Adobe User Group does it again! And how! At the threshold of end of the year, on the 3rd December, the dutch AUG in co-operation with Adobe presents to you a full XL event. With the popularity of the normal meetings rising and rising and these event being of an exceptional level for any type of usergroup, this time, the lid is fully blown off!

The XL event will bring some great national and international speakers to the stage; Dave August of OOOii will talk about his use of Flash in JJ Abram's Star Trek, as well as his creative input to movies such as Fast & Furious. Tobias Mannewitz of Karakter will be there and with work on projects for Spellbound, EA Games and SCEE this promises to be a session not to miss! Anita and Geoff from Champagne Valentine will amongst other great things talk about an interactive music video they created for Placebo.  Adobe will give you insights in their products and tooling with such international renowned speakers / evangelists as Serge Jespers, Jason Levine and Paul Burnett. If that's not enough, Niko Stumpo of Hanazuki will be talking about the studio's new work together with Mark Barzinski of Barzinski and Jean Jean. Dirk Groten of Layar fame will be talking about the Layar platform, something I'm looking forward to find out more about. Last but hopefully not least, I was honored to get an invitation to speak, and I will be talking about my latest forays into realtime flash.

This lineup is already stellar, with more names to come and be added to the list! Combine that with the friendly atmosphere and a great place to extend your network, learn and enjoy, this is an event not to be missed. Even if this is run by the dutch usergroup, I wouldn't be surprised if people come in from all over, as this XL event truly is XL. Be there! And if you do, make sure you say hi!

Get more information at the XL webpage, follow the AUG on twitter, and order the insanely low priced tickets here.

No related posts.

Last week at Flash on the Beach I had an announcement regarding me and Papervision3D. While some people took that as a hint to the release of PapervisionX, the actual news was a bit more sad. After pondering on this for a while, I made the rather big and tough decision to leave the Papervision3D team. As many of you couldn't be in the session, since the room was packed and as I heard later, many, many people not being able to come in from the line outside. Next to that many of you not able to come out to FOTB, this post is to ellaborate on my decision and explain it to all of you.

A great experience

I've been on the Papervision3D team for around 3 years in total. Only weeks after it's first closed alpha release, I joined the effort. All in all it was a great ride and I had an enormous amount of fun working on the project. Mind you, contrary to what some people believe, Papervision3D is not a company, there is no steady revenue and there is no direct income from it. It's an opensource project, with all the team members contributing out of fun, love and a feel for our great community. With the team, we have been able to deliver a solid, fun and easy to use 3D API for the Flash Platform. While Papervision3D certainly wasn't the first Flash 3D engine, many of it's features where and it's ease of use propelled the name to the top-ranking name in Flash 3D engines.  Along with that, Papervision3D helped putting Flash 3D solidly on the map for companies and individuals, making Flash 3D a tool accepted and available within the creative palette. I'm very proud to have been part of that and very grateful for all the valuable and nice feedback from the community. When I started on the project, it was only me, John Grden and founder Carlos Ulloa working on it. With time the team grew, changed composition and we also got more and more community input. These days, the team has grown to a rather large size, we got many blogs and sites dedicated to the engine and a vibrant community working with and on the engine. There's nothing to describe how proud I am of achieving that together with my fellow team members.

Building engines

Over time, I've spent more and more time in creating and architecting the core of 3D engines. With Papervision3D, this was the 1.7, 1.9 and 2.0 core of the engine. For PapervisionX I've taken that even further, creating the entire core architecture and implementing the many things I've learned over the years. But next to Papervision3D, I've also built multiple engines in my commercial effort as a freelancer, and have greatly enjoyed doing so. During my FOTB presentation, I showed a video of the Anne Frank Virtual House, for which I was commissioned to build the 3D engine by LBi / Lost Boys and the Anne Frank Foundation. For the project I worked closely together with Floris Drupsteen of Biqini, the 3D modeller on this project, who did an awesome job of building the actual 3D models. It was an all in all great experience to build an engine on that scale and I'm proud to have been part of that effort. Considering the goal of the effort, it was a very meaningful implementation of a 3D engine to me, and an honor to be part of that effort.

With me stopping to work on the Papervision3D project, this is certainly not the end of me building 3D engines. Rather a rehash on how and why I do that. First and foremost, I will keep in chipping in the community effort, through this blog and upcoming projects to be released in the opensource domain. Also, commercially, I see myself building graphical / 3D engines, when possible, for a while to come too. All in all, I just love building the graphical engines for projects, hence also my self-given titel; "visual programmer". With no noticable big difference in what I do, there is a question remaining.

Why ?

There's a couple of reasons to the exact why of me leaving. First and foremost, this is a personal choice on my side, has nothing to do with my feelings towards the engine or the other team members. An important factor for me, things have to do with momentum. The engine has had an incredible amount of momentum over the years, and that pushes me to work on such a project even harder. Currently the project is a bit lower on momentum, which doesn't help me either. I think it's important to know I've been working on the engine many, many hours over the last years, and at some point you get to some fatigue of working on the same thing for a while. With the success of the engine and the personal success of my fellow team members, all of us have gotten more busy on the commercial side of things. This means less time goes into the project, which in turns into less momentum for the entire project, which in turn leaves me less "pushed" to do more. Considering the enormous amount of pressure from people on the outside, this wasn't good for me either. On the other side, this change in team composition, with me leaving, gives room for other people to full-fill that position and get new ideas, new work and new momentum going. All in all, I think it will do the engine good.

Next to momentum, another important reason for me was the goal of building an opensource 3D engine. With many people doing awesome experiments and examples / implementations of the engine, it really gives me a natural high to see how it's beeing used. But next to that I'm also looking to create beautiful things myself. Now, I'm not anything near a real visual designer, I like to see myself as one who builds the tools and technology to help enable these experiences. But I want to be part of something which will lead to beauty in visuals, story telling and emotion of the experience. Without a doubt many Papervision3D powered experiences can be classified as beautiful experiences, but most of the time, just building the engine means that they use the technology, but you are not directly involved in that process of creation. I really feel I need to work on projects which will give me a more direct link to the artists, or even more closely enabling artists. One of the projects I have recently joined is the Rhonda Forever project. Certainly very different as a tool to people, my job within the team is very similar to what I did for Papervision3D. For Rhonda I'm building the online drawing viewer for the project. The change for me there is that I will be working directly with it's creative community and seeing results of that effort in a more tangible/graphical way.

Wanting to create visual beauty also means that for me, there was a decisive direction to take. The work you do is the work that defines you within the industry and that in turn gets you the next big job. With me wanting to do more directly involved work on creating things with beauty, but my work mostly being hidden in technical solutions like for instance Papervision3D, I wanted to expand my working area to more visual beauty and become more part of that side of the process. The only way to do that is by just start doing it, finding people to work with and changing that direction. This one of my first steps towards that. With a renewed sense of freedom, I'm hoping to work on projects more closely and be part of the actual implementation more, rather then being part of a tool that enables people to do that, but not being part of the creation of the end result.

Community wise; I'm still there. I am pondering on some of the directions. Along with doing more work with and on OpenFrameworks and Rhonda, I will still build 3D things for the Flash Platform. What I can release, will be released through this blog. With the 3D Flash community having considerably grown from near to nothing to an almost distinct part of the industry, I will be part of it. Also, I'm looking forward to build technology which spans and extends all the available 3D engines, rather then just focusing on one engine.

PapervisionX

One thing I'm not entirely sure of as of yet, is what to do with the PapervisionX code base. Built entirely from scratch by me and Tim Knip, this project really has part of my soul in it. It's been exclusively me and Tim working on it, with plans to have the rest of the team start on it a while back. Currently, that hasn't happened yet and the engine is not ready for release and officially not part of the Papervision3D code base. In other words; it's mine and Tim's. While I'd love to release it at some point in time, currently it's not ready for that and with me leaving the team, it's most likely not going to be released under that name, if released at all. I might decide together with Tim to donate it to the Papervision3D project, but currently I'm not sure on that decision yet; as there would have to be a considerable effort to make it stable for release. As it is unfinished now and publishing source code is somewhat a personal thing when most of it is yours, I'd think this is not what should be done right now. As I am going to pursue my own goals for the time being, this decision might take a while.

I will discuss with the team and Tim closely to see what can be done in the future, but for now, I don't think it should be released. It might also be better for the team to work on it's own Flash 10 compatible version of the engine.

Triangle3D

Another engine I showed at FOTB was my personal experimental 3D engine Triangle3D. Not at all meant to compete with any of the 3D engine, rather to be my own basis for experimenting with Flash and 3D, this engine might be released through this blog in the nearer future. Again, this is not me splitting off from the team and doing my own thing, this is more or less just giving the community my personal playground for Flash 3D. The engine is far from feature complete, and honestly, I don't want to focus on that. Rather just build it as I go and need something. This engine will most likely form the basis for my commercial 3D efforts too, as a custom based engine is almost always going to be more focussed on the specific implementation then a more generalized 3D engine API.

Thanks

All in all, in regards of the Papervision3D project, I would like to thank my fellow team members for all the good times. I would like to thank the community for their beautiful implementations and experiments as well as feedback. You make it all happen! Lastly, as a special member of the team, I would like to thank founder Carlos Ulloa for having me on board in the first place. It's been a great ride, and I'm sure we'll be riding our race cars for quite some time, even if we have to build them ourselves.

Related posts:

1. Flash Player 10.1 pre-release on Nexus One Nexus One running Flash Player 10.1 from Ralph Hauwert...

With the addition of the 2.5d API in Flash 10, lot's of developers now have even easier access to basic 2.5D environments. But with those new capabilities and API's new challenges for developers arise. Famously, Flash 10 doesn't do the Z-Sorting natively. When Lee Brimelow asked me if I could send him an easy solution for that, I sent him the SimpleZSorter. Which he in turn used to paste my profile picture on as much as possible planes, but more significantly, use for this tutorial.

Althought the name Simple would imply something a bit away from a PerfectZSorter (which, technically, has some challenges considering the 2.5D API and the player performance, don't ask), the thing which astounded me was the amount of feedback. Sure, more people had asked me for such a solution and this simple little tool did it for them most of the time.

But considering the simplicity of the SimpleZSorter and the amount of feedback I got on it (even if I didn't publish on it myself) made clear that although the 2.5D API of Flash 10 opened up loads of opportunities for developers and designers a like, some of the simpler issues aren't addressed by it, whist there is clearly a need for it. That need can be addressed and with this post I'm hoping to do a another small part.

The ArcBall

One of the questions which people have been asking me about for in relation to using the 2.5D API, is (loosely combined) "How do I rotate things in 3D correctly with the mouse".As this question came more and more, I've also realized how fundamental this is to our community. With Flash 10 we have native realtime (2.5D) 3D, but with realtime comes interactivity. And what use is it for users to have a 3D interface, while the controls are not intuitive ? At that point 3D becomes a worse experience, rather then a better. While this solution won't address every issue with that, it most certainly addresses one, 3D rotation dragging.

Native 3D interaction might be new to Flash, but it certainly isn't new to Flash 3D, or 3D in general for that matter. And the problem of having a user rotate an object with a mouse has been addressed before in numerous different ways. So luckily the wheel doesn't need any inventing. The wheel I chose for to address this problem is called an ArcBall, first proposed by Ken Shoemake.  Now if you want to read that paper go ahead, but let me define this in flashers layman terms.

The Problem

Let's first have a look at the problem as I've seen examples of code of people and derived it to this "problem" example. Drag the cube to rotate it.

Although the mouse dragging does offer some feeling of control, you'll find it difficult to rotate it in a natural intuitive way, to let's say to the front of one of those colored planes. Here's what's happening here in that code, and what I've been seeing most.

if(isDragging){
boxSprite.rotationX = mouseY;
boxSprite.rotationY = mouseX;
}

This is example is the most common and is responsible for the example above. Now, the problem here is that it doesn't take into account the local object transformation, as well as the dragging coordinates. As the object rotates around it's axis, so do it's axis change. Also, as the dragging goes on, we don't take into account the 2D coordinate versus it's 3D transformation dragging start and final output. All in all it ends up being a bit messy.

The Solution

Here's the solution using the ArcBall class. Drag to rotate.

The solution, the complex part

As you can see / feel, object rotations now feel much more direct and natural. But what is this magic ? First off all, the class is fully Flash 10, and uses it's native methods and features, where available for this solution. To explain it as simple as possible, as you click and start dragging on the cube, a sphere is established. This virtual sphere, around the object is then used to establish a point in 3D space on the sphere. As you are dragging, your actually rotating other points on the sphere. From these angular differences on the sphere, we can derive a new transform matrix for the object.

The hard part is in the establishing the rotational angles from those 2 coordinates on a sphere. A well proven way to do that (and the only form I have seen in ArcBall implementations, so far), is using Quaternions (go ahead, read it now, all of it, I dare you). Quaternions are complex things (pun intended). Now Quaternions are a bit daunting to start at and at this point I would be lying if I said I fully understand them. I'm not the only one who finds them scary, quoting Keith Peters / Bit-101 ; "Oh no. Vector3D’s w is the second most scary w in the world." (I wonder what the first is, Keith ?).

Quaternions and Flash 10

If you are looking for the clearcut solution and implementation of my class for it, skip this part, otherwise, read on, enjoy.

The reference to the w in that quote is also a reference to the complex component real component of the quaternion, where the xyz are complex numbers. Yes, complex numbers (I dare you again) and their appliance is where stuff does get scary for most of us. The great thing about the Flash 10 3D and geom API is that there is a partial native support for quaternions, next to having a vector with x,y,z and w components. For instance, this:

matrix3D.decompose(OrientationStyle.QUATERNION);

Which will return a Vector of 3 Vector3D's, which are in order translation, rotation, and scale. When using the quaternion orientation style, the rotation will be filled with a Vector3D with it's x,y,z,w values nicely filled; a quaternion. While this is something we could have handled ourselves (decomposing a matrix to it's components using quaternions), it's great Flash 10 has it. Less code, less size and it's faster then doing it with your own defined functions.

This ArcBall implementation relies heavily on that piece of code and some added code to do some additional stuff with the quaternions (multiply doesn't seem to be available native). Now while I'm not enough of an authority to try and explain quaternions, the applications for them in 3D rotation are numerous, most famously, the first thing which will get mentioned when talking about the topic of Gimbal lock, which is a problem most encountered by astronauts and people developing 3D applications (see! us developers do have something in common with astronauts, we're now doing rocket science!).

That being said, on a wider look on Quaternions, for example, Both Andy Zupko and John Lindquist have done some nice articles on the subject and it's appliances in Papervision3D. Papervision3D, running from Flash 9 and higher, has it's own Quaternion implementation, as Flash 9 doesn't support it natively.

Deriving an easy solution for all

While the solution to ArcBalls using Quaternions isn't even that hard, I still felt the problem should be abstracted a bit more, with an easy one stop solution for all. After all, the reason I got that much feedback on the SimpleZSorter, is clearly because people actually use it. Why do people use it ? My estimate is, it addresses a real problem and it's simple to implement. And while that might cause a little overhead, and make it less flexible, it works for people looking for a simple solution. Here's how that was implemented.

SimpleZSorter.sortClips(boxSprite,true);

Simple enough right ? So for the abstraction of the ArcBall something similar made sense.
Using the ArcBall class (and getting your correct dragging rotations up and running in no time) works like this.

arcBall = new ArcBall(boxSprite);

Now you can drag this object, and get nice, intuive rotations from it. There are a bunch of options to use;

arcBall = new ArcBall(boxSprite, useRadius, radius);

If you set the useRadius argument to true, and set radius to a specific value for that object (think about a sphere which completely encapsulates the object you have, it's bounding sphere, see my beautiful artistic rendering at the top of this post), the bounding sphere used per dragging action is the same size. When that option is left set to false, the point on the sphere is derived from the click in 3D space on that object and thus speed might vary, depending where the user start the drag.

Because the ArcBall class automatically adds listeners to the object, and automatically enables dragging on it, you need a way to switch it off when it's not wanted.

arcBall.enabled = false;
//To enable it again
arcBall.enabled = true;

So, that's it. I hope it helps you in getting a better experience for the end user. Since this is an abstraction for ease of use and much more can be done, the entire code base for both examples and the ArcBall under the MIT License; essentially, do with it what you want, if it breaks anything, it's not my fault. Let me know what you think.

Lastly a mention of sources which helped me with getting this right; rainwarrior, NeHe, processing. Sharing works!

Sources

ArcBall (just the sources for use).

RotationWrong (sources for the first example, incorrect drag rotation)

RotationArcBall (sources for the second example, correct dragging using the ArcBall class)

[Update 8/9/2009]

Little update on the quaternion explanation.

No related posts.

I normally don't delve into book reviews or endorsements; and I won't go there just yet with this post. But I do want to get your attention for the first released Papervision3D book; Papervision3D Essentials.

Written by Jeff Winder and Paul Tondeur, this elaborate, 13 chapter book looks great for anyone wanting to learn or deepen their knowledge of the Papervision3D API and Engine. Congratulations to both authors for getting it released! Great job!

Although I haven't recieved my ([total disclosure] free, thank you guys), copy for review yet, the book looks promising, if you can judge it by these two free online chapters, about Lines3D and VectorVision.

As one of the core members of Papervision3D and having architectured and written rather large portions of the Papervision 2 Engine, for me personally it's amazing to see a project like this. It's documenting code that the team has been writing over the years and getting it on paper, along with an explanation and examples is something huge to me, as well as being huge for people wanting to start or already experienced with the engine! Having recieved multiple mails full of questions and additional bug fixes (as have other members of the team) from Paul and Jeff, I can assure they have been thorough in reviewing & examplifying the Papervision code base.

In a way, that's a very odd feeling; an obscure function which might have been thought up and written at 2pm at night (which during the initial dev of Papervision3D 1.7, 1.9 and 2.0 was rule, rather then exception), might now have gotten an explanation in the book.

One thing the project has been lacking the most though, was a paper resource to look things up; and that has now arrived!

I will blog a more thorough review after having recieved and read the book. Can't wait to see this book as a piece of tangible material and actually hold it!

[UPDATE]

Another free chapter has emerged as PDF. Chapter 8; modelling and importing to Papervision3D.

No related posts.

Microsoft released Silverlight 3. So it's time for Silverlight on my Flash blog ? Heresy ?  No, not really.

I'm not going to do an intro to Silverlight here, I guess you all know what it is, or what it is proposed to be. As a developer, I'm naturally interested in platforms available to me or in a similar area to what I work in. Therefore, Silverlight has my attention. Not because I believe it's a Flash killer, not because I think a bag of Microsofts cash will come my way, not because I want to destroy Silverlight, but because it's a technology which plays in the same realm as my professional environment. Surely being Flash Platform developer, I'm biased. The platform that has allowed me to earn my professional living and have so much fun is clearly my favorite any day. But let's have a look anyway.

Silverlight vs Flash

I've always found the developer oriented marketing message for Silverlight somewhat confusion. One moment it has nothing to do with the realm of Flash, the other it's the designated Flash killer, and I find myself hosting heated discussions from both camps. It's been confusing. Personally, I've had a look at SL1, 1.1 and left it there. During those tests I found it was still very young technology and a far cry from the capabilities from my personal favorite platform, Adobe Flash.

I remember the first release site for Silverlight, and it left me confused if I was watching a commercial for a snowboard brand or a new web technology. Things seem clearer now, Silverlight clearly has an ambition towards what Flash is, but currently still isn't. Neither towards end users or developers. What has happened with the release of SL3 ?

Features

Now in it's 3rd iteration, the Silverlight team seems to be playing catch up and there's an akward similarity between Silverlight 3 and Flash 8/9/10. Let's look at a selection (these are the biggest though) of the features added, and how they translate to the world of Flash.

I'd like to get one thing out of the way as an older feature; Silverlight 1.1 - .NET support / Translates losely into ActionScript AS2 to AS3. A far better runtime for code in both cases.

- Silverlight Perspective3D - Flash 10 Perspective 3D
Well, with both players now doing 2.5D "postcards in space" natively, it's up to third party developers to build the 3D frameworks. Good thing I like that kind of stuff a lot. Before, it didn't seem possible to build anything like Papervision3D on a usable level for Silverlight, but I think that with 3.0 we might finally see some competition coming from the Silverlight camp, now that they can actually control bitmap content ?

- Silverlight PixelShader API's - Flash 10 PixelBender & Flash 8/9 Bitmap Effects
I'm not entirely sure if Silverlight PixelShader API's are GPU accelerated or not. Flash 10 Pixelbender surely isn't. So far I haven't seen a running version of Silverlight PixelShader API yet, but hoping someone will fill me in. The ability to write 'real' HLSL for it seems enticing. Currently I don't think there's enough demo's online on the Silverlight side, but it would be interested to see where that goes.

- Silverlight Cached Compositing - Flash 8/9 gpu assist (fullscreen, mac) and Flash 10 GPU / Direct WMODE.
Somewhat similar to each other, Silverlight Cached compositing seems to be a combo of using the GPU for rendering as well as good old, cacheAsBitmap. I'd be interested to hear from both camps how far integration goes. From a developer perspective to both platform, both seem to just be a "switch" to be turned on, no control.

- Silverlight Bitmap API & BitmapCaching- Flash 8 Bitmaps and .cacheAsBitmap
Silverlight gets my favorite Flash 8 feature. Bitmaps. WriteAble bitmap is similar to BitmapData and allows for richer effects and graphics to be created by the developer. BitmapCaching is surprisingly similar in even name to .cacheAsBitmap, which allows the developer to reduce the overhead of rasterizing components of which the visual state has not changed.

- Silverlight LocalConnection API - Flash 6 LocalConnection API
What's there too say ? Player instance talking to Player instance. I actually assumed this was already inside Silverlight, but hey, now they do. Seems to be related with Silverlight endeavours on the Desktop (think AIR, but a little less well featured).

- Silverlight Application Model Enhancements - Flash 9/10 FileReference API
Again, this seems to be related to MS preparing Silverlight to jump to the desktop, although this feature is available in both Flash and it's desktop brother AIR.

- Silverlight Assembly Caching - Flex Framework / Flash 9 RSL Caching.
Storing chunks of frameworks on the client so they don't have to be loaded in everytime for every site / platform. Not sure how cross platforms silverlight solution is. Neither do I know how to erase that cache with SL3.

- Silverlight Network Monitoring API - AIR ServiceMonitor, SocketMonitor and URLMonitor
Again, a feature for SL jumping to Desktop, this allows you to see if you're connected to the cloud or not. AIR is a little more extensive with it, Flash has no implicit features.

- Silverlight Application Frameworks - Seems to be a code base like the Flex SDK
Application Frameworks loosely relates to what we call the Flex framework, and the enhancements on that for SL3 seems be very minor. Flex2 - Flex 3 - Flex 4 (gumbo) steps seem much bigger.

- Silverlight UIFrameworks Enhancements - Flex SDK / Flash Components
Well, not much to say here either, UIFrameworks seems to be playing catch up it with Flash / Flex UI Components (the latter to be most precise).

- Silverlight Text Improvements - Flash 8/9/10 text improvements
Although the new Text Rendering engine for Flash 10 is a huge step, the Silvlerlight of things looks more like the new control over text rendering like the one we got in Flash 8-9. Flash text layouting seems to be far ahead on this one.

- Silverlight Animation Easing Effects - Flash - Robert Penner, anyone ? Combined with some mx.easing.
Again, this was a feature I thought to be native Silverlight already, but it is just introduced now. As with the early days with Flash easing, I hope people see how to leverage this for a better UI, and not an experience of windows and things shooting by, rendering the UI useless.

A strange coincidence....

A lot of similarity between new SL3 features and Flash 8-9-10 features, right ?
Now, none of this is a coincidence. All these features are surprisingly similar to some of the new features we've seen appearing in Flash over the last 4 years. In a way, you can say SL is still playing catch-up. In the new feature set I see no real innovative RIA / Plugin features that really distinct Silverlight. That being said, SL plugin is still not as feature rich as Flash 10 is currently. Seeing what Microsoft has done in 3 versions, one could asses that they are moving very fast on the plugin feature front. But there are some things consider here. First of all, Silverlight obviously is learning it's lessons and features from Flash. I'm yet to be amazed by a novel Silverlight feature. Another thing to consider is that Silverlight 1, 1.1 and 2 compared to Flash had virtually no installed base, and backwards compatibility of all features and API's doesn't seem to be retained. Compare that with a Flash Player with more then 10 years of history and still be entirely backwards compatible.

Plugin stability

Stability wise, my experience with the SL plugin have been less then good. Browser crashes, followed by content not running (I'm still looking to see working demos of most of the new features online, which run on my mac too, anyone ?). The experience of years of Flash Player seems to be owning Silverlight here. With content behaving differently between my both Macbook Pro's, ubiquity doesn't seem to be the strong point of Silverlight; I'd say this would be me only, but my twitter feed tells me differently today.

Video

One thing which is worth mentioning in Silverlight's favor is the accelerated video rendering. Flash 9 / Moviestar (9.0.115) introduced fullscreen acceleration to allow for smoother feedback, but the acceleration mentioned for Silverlight seems to be decoding based. VC-1 is supported by graphic card manufacturers, so this would mean the actual decoding is done on the GPU, allowing the plugin to offload that work from the CPU. In that same area, DRM for VC-1 seems to be pretty sturdy too.

Tools

On the tools front, I can't say that much, since the Microsoft tools for building Silverlight content still don't run native on Mac OS. From back in the day, I can still say the Visual Studio IDE is one of the best Microsoft products, but beyond that, I have no experience with their new tools. I hear integration with Photoshop is pretty good. In a way tools and development language will also greatly influence the people using them, in turn influencing the output created for the end users. Flash is still a very nice melee of designers, actionscript developers coming from a design backend, and hardcode java developers turned Flexer (just to name a few sub categories). At any conference we always get a nice melee of peope all with their own angle and usage of the platform. This really reflects in all the different content being produced for the platform, where I'm still waiting to see that diversity in the Silverlight universe.

Some personal ideas

Now, although SL is making some progress on their own feature front, they are still behind. But in the end it's about the end user experience, and this is where Microsoft clearly failed in this context. Their see the light website is less then optimal, if I put it mildly. Seemingly infite loading time, followed by a horrendous design and user experience, leaving me wondering where all those megabytes of preloaded content went.I wonder if this is a trend amongst all SL content, as I've yet to not see one of those dead ugly SL preloader circles. This is the site which is supposed to showcase Silverlight, but it is it own worst advertising. Maybe they can get a Flasher to redo the seethelight.com website.

[UPDATE 11th July 2009]

Apparently, I'm not the only one who didn't find seethelight.com up to par, after seeing the light, microsoft removed it, linking directly to the html version (in the Silverlight version, this was linked as "reduced experience"). Now, we get the "reduced experience" by default. Let's hope they come with something to replace it, and be a real showcase for Silverlight.

[/UPDATE]

Silverlight is good for Flashers too.

All in all, Silverlight is good for everyone. It means competition in the field of rich internet plugins. Yes, there are more. But these are the 2 big companies playing and competing in that field. (I really don't think Sun is a real contender in this area yet). And general logic would say that with competition, both products will be more rapidly developed and improved. Which, for us developers and end users is only a good thing. With SL3 being just out, we'll just have to see how many developers adopt it and how many content providers are willing to take the jump on a non-ubiqitous plugin and how steep the adoption slope will be. In terms of catching up,in my opinion it's still Microsoft's move (even with SL3), but the incremental step they did for this version looks promising for the future. At which point we can all look forward to Flash 11, Flash on devices and mobile to compete with that.

No related posts.

I just stumbled upon this very short TED talk by Chris Hughes. In his presentation Chris Hughes shows of an "Augmented Reality demo using Adobe Flash!". In an exited tone he talks about how he wrote a piece of software (!) to allow the replacement of a "2D Barcode" with something "really, really cool" using nothing but Adobe Flash!

In reality he hardly wrote any software, rather did he do a pretty standard and in my opinion a very ugly implementation of Papervision3D and FLARToolkit, for which tutorials can be found online quite easily.

Why is this worth blogging ? Well, as a Papervision3D core developer, personally I'm offended for someone taking credit for both the Papervision3D and the libspark project's FLARToolkit libaries, none of which he has had any hand in. He's essentially taking credit for 100's of hours of work by both the Papervision3D and the Libspark team. With the TED conference headline being "Ideas worth spreading", which implies the sharing of knowledge and ideas. Both Papervision3D and FLARToolkit are opensource projects, with exactly that in mind. And both of them get ripped off by Chris Hughes, who not only fails to cite his sources, but takes credit for it by saying the following things on the TED stage :

"......I wrote a piece of software ...... "

"Oh, no, it does not quite stop there.......[]......another bit of the technology we managed to introduce here......"

With both statement not only failing to mention the actual technology and projects, Chris Hughes but actually taking credit for it. Disgusting. But hold on, to cite Chris himself, "it does not quite stop there"....in the followup interview on TED.cm Chris Hughes states the following :

"So what you've seen today is a Flash port of an existing project. There's a project out there known as ARToolKit. Unbelievably awesome. It's a university research project. Folks ported it from C, which is what it's written on, onto Java. And then we took the Java port and made it work in Flash."

That last piece of that sentence is very interesting "we took the java port and made it work in Flash". Let's make it easy. No you didn't. You took a port named FLARToolkit, and used it in your project. (here's the sources to his demos, which show Papervision3D and FLARToolkit).

So what did Chris Hughes do for him to take credit on a TED Stage ? Most likely, he followed a tutorial like this one from the FlashBlog, then gathered all his courage and energy to work with 2 opensource projects and take credit for it.

One more quote from Chris Hughes here :

"It's open source. Everything I do is about being very open."

Openness at a conference like TED would be giving urls, credit and a clear explanation of what you did. Right now you're just taking full credit for work done by 2 software team's hundreds (if not thousands of hours) of work, for your own credit.

I hope the TED conference takes the proper steps to avoid people like Chris Hughes standing upon on a stage, presenting other people's work as their own, purely for their own glory. I'm disgusted by this.

[EDIT]

To avoid any confusion, looking back at this hastly and angrily typed post, I want to clarify that most of the work he's demoing which makes this so awesome is the FLARToolkit project. Papervision3D is used for visualization, but I'm guessing most of the awe is for the FLARToolkit project, which is done mainly by saqoosha.

[UPDATE]

The TED conference has removed the video from the Talks page, and is in the process of removing subsequent posts, as a response to the contreversy around Chris his talk. I do feel a rectification of the statement "And then we took the Java port and made it work in Flash." is still missing, since this is just clearly not true.

I have a lot of feeling towards this kind of presentation of other peoples work without proper accreditation and honestly, I do feel the real victim here is Saqoosha / LibSpark. Saqoosha has posted a blog around this, unfortunately only in japanese.

I'm sorry for the uproar and feeling some people have of this being a witch hunt, or the feeling this was instigating "get your pitchforks and torches out". That was not the intent of my blogpost. I feel that injustice was done, and it wasn't publicly rectified. As this blogpost was part of the reasoning behind the removal of the video of the TED pages, I do feel it served the proper purpose and in that specific result hit it's goal.

It wasn't my intent to be insulting, or publicly binding Chris Hughes to a stake. Rather this was a way to rectify the injustice which was done to the original developers of the source code he displayed, and took credit for. I apologize for any of my wording which is that strong above and as such removing it would be wrong. Rather then doing that, I'll leave it up and apologize for too strong wording or instigation of a witch hunt. What I won't apologize for is pointing out this issue and subsequently asking attention for it.

[UPDATE2]

I have just been in an enlightning IM discussion with Keith Peters / Bit-101.  In this discussion Keith pointed something out I feel I am at fault at. I've could have contacted Chris personally to ask him for a public correction of both his talk and the interview. Instead, I first wrote this post and then mailed Chris later, after any reply from his side stayed out.

I've got no excuse for this and I am at fault for not contacting him directly first. I apologize for that towards Chris Hughes.

[UPDATE3]

Upon further investigation, TED.com have decided to completely retract the entire speech, naming it "misleading".

[UPDATE4 25/06/09]

With TED having researched the grounds of these allegations and subsequently removing the video and retracting the entire thing, Chris Hughes still wants to pass this off as a fluke. I have yet to see an updated blogpost, or a reply to my emails. As such, let me make it perfectly clear for the last time, for future reference.

Chris Hughes took Mikko Haapoja's tutorial on using FLARToolkit and Papervision3D, two open source projects he was not involved in at all. Then he did some slight modification of the tutorials, not adding, or extending anything and recompiled the examples. He then went on stage saying "He programmed" etc...

In reality, he passed someone else's work off as his own, without giving attribution or even mentioning the projects, rather just claiming credit for it in an interview, and obfuscating what he did both online and in his presentation.

End of story.

No related posts.

Before I start, I want to explain the timeline which led to this blog post. This was supposed to become a post of being able to get much faster access to bytearray's through ActionScript, by (ab)using the Alchemy memory opcodes. I spent quite some time trying to build a swc library which would allow you to do just that. Only to learn that you can't build a generalized API for this in ActionScript. It just can't work out with the current ActionScript compiler technology; here's the story of why.

Leveling the field between Alchemy, Haxe and Actionscript

The reason I set out to do this, is because I found it somewhat strange that the only language in the row above that is not able to access the faster memory, is the platform's native language, ActionScript. I could use many strong terms for this, but it is what it is. So why not change this myself ? Surely adding a piece of custom built (byte) code, should be able to allow access to this hidden gem ? I had discussed this option with Joa before and it seemed to be a viable path. So why not do it ? A good thing to note in the mention above is that Haxe has already implemented this. Read Nicolas his post about haxe flash.Memory.

Compiling inline ABC with Alchemy's asc.jar, a bit different.

As I posted earlier about Alchemy, one of it's advantages is the fact that Flash Player 10 has an implementation for actionscript byte code opcodes that are currently not supported within the ActionScript API. These opcodes allow for faster access to 1 bytearray inside the player, which you can set by accessing ApplicationDomain.currentDomain.domainMemory.

Strangely enough, that is actually opened up to AS API, presumably to allow for "regular" ActionScript access to the Alchemy memory. The actual opcodes have no ActionScript equivalent though.

Since Alchemy boils C code down to ActionScript, how does it obtain access to these opcodes ? The answer lies in the compiler which comes with Alchemy. The asc.jar in the /bin package allows for so called "inline asm".  This type of code looks something like the following.

__asm(op(0xD2),op(0xD1),op(0x3A));

This code can be put anywhere, and the opcodes will be added to the abc bytestream. Great huh ? So, ideally, we would just switch out our asc.jar in our Flex install directory, and we could just start doing this. Well. NO.
Compc, which is used for actionscript project compilation isn't compatible with this special asc.jar. Not too worry, we can get around this, more on that later.

Discovering special memory opcodes
Although at the first release of Alchemy, these special memory opcodes where somewhat ellusive, by now there's loads of information on that. Most specific info comes from Adobe Alchemy's creator, Scott Petersen, in his presentation FlashCCompiler (pdf).

Now, armed with these special opcodes, writting an API around it shouldn't be to hard, so let's go at it.

public function setFastMem(byteArray : ByteArray):void
{
 currentDomain.domainMemory = byteArray;
}
 
public function writeByte(position:uint, value:int):void
{
__asm(op(0xD2),op(0xD1),op(0x3A));
}

Ha, yes, combined with the Alchemy asc.jar, this code works, and translates out into working bytecode! So, now by calling writeByte, we are using one of the opcodes to write an opcode into the FastMem. Great! When I had all the opcodes to work, I forgot to do one thing, which I should have known and could have saved me a lot of time. Benchmark it. Instead, I first wanted to figure out a way to distribute this, so developers could easily do this.

The long path from .abc to .swc
Now, when using the asc.jar, we can not compile a swc directly, since the asc.jar is incompatible with the compc compiler.
It's only able to output a swf, or the abc bytecode for that class.
What I went and did was examine the gcc script in the alchemy/achacks folder. Interestingly, the gcc script is a replacement for the normal gnu gcc compiler, written in c. It handles the steps for alchemy to go from c to llvm to actionscript to swf/swc distibutable. Even more interesting is HOW alchemy creates a swc, if you use the -swc compiler option. You'd think there would be some magic going on there, but instead it is not so magic at all. When compiling to a swc file, the compiler extracts the abc data from the previously compiled swf using a script called GetABC2.pl, and then replaces the code in an existing .swf file which is in the achacks directory, using PutABC2.pl.

A thing to note about that .swf file it uses, is that the file is an uncompressed .swf, marked in the header by FWS. These are getting more and more rare, since most swf files are compressed these days, and start with CWS instead. This is mostly common knowledge. Now, the problem here is that I wanted to replace that template.swf by my own....only to find out there's no documented way of compiling a swf file without compression. To make matters worse, the perl helper scripts seem to have a bug, where they do not work with compressed files. Although these scripts can be patched, that's not what I wanted to spend my time on, so I went my own route.

Injecting bytecode
Instead, I wrote a little air app, which scanned a swf for the DoABC tag, (id 82), and replaced it with my own ABC bytecode. This involved changing the size tags of both the swf and the tag, and saving it out again. It took little time, but in the end it worked like a charm, and I could now generate my own swf's with custom inject ABC bytecode, great! Now I needed to get this swf into a .swc file.

SWC and zip, a little tip
As most of you would know, as SWC file is nothing more then a .zip file, normally containing a library.swf and a catalog file. In my naive thought I would just unzip the swc, replace the library with my abc injected library and things would work. Not so. I'll save you the hours of pondering on this, but as it turns out, when using the Mac OS X finder, and unzipping it through that, leaves you with something crippled; what I don't know. So here's a good tip, if you want to unzip and rezip a SWC, do so using the terminal / unzip & zip. Really, I wish someone had blogged about this, because it drove me crazy, ending up with Flex Builder saying "can't load swc", even if I didn't change anything in the .swc, just uncompressed and compressed it again. I caught on to that quite fast, but the fact it was the uncompressing the swc which was the problem, took me a little while to realize.

It works! Now let's benchmark
As I said before, with some logical reason and benchmarking before I went on a spent a lot of time on bytecode injectors, build scripts and swc compilation, I could have saved myself a lot of time. The reason why I wanted to have this distributable was the entire reason to do so, so I didn't want to go on, without knowing how to get from a special __asm .as file to a working .swc. In the end it was all in vain. When I started benchmarking this, the result was disappointing.

The FastMem bytearray, and the special functions I had built for it, where nearly 2x SLOWER then just using ByteArray.writeByte(). And then I figured it out. I have forget to think about the overhead of calling a function in ActionScript. Since a built in function is much faster then an actionscript function, the overhead of calling that function far outweighed the faster access speed to the data.

So it ended up in a failed experiment. That's ok. Failing is a way of learning, and during this process I learned a lot of new things. But in this case, the biggest lesson wasn't how to inject bytecode, use the asc.jar, write your own swc compilation pipeline. It was that with the current compiler for ActionScript, it's just not possible to use the FastMem opcodes in a easy and generalized way.

The difference between the compilers

Now, the saddening conclusion is that I wasn't able to build a FastMemory.swc that all Flash developers could use. The more alarming conclusion I take from this, that the problem doesn't just resides in the compiler not having any way to do so, but the fact that it's due to missing features other compilers already have, leaving ActionScript developers behind.

Haxe is able to pull the exact same thing off. With Nicolas his work on the flash.Memory api, you can select a bytearray, and get very fast read and write access. So why isn't this possible with ActionScript ? The Haxe compiler supports inlining. For those unfamiliar with this, this allows you to keep portable and reusable functions, while the compiler inserts these functions directly in the place they are called. So you're not actually calling function, you are essentially inserting a piece of code inside your code. This eliminates the overhead of the function call during the compilation phase.

The reason this does work within Alchemy is that it not only supports inlining (less important in this case), but the Alchemy code actually is processed before hand by the llvm compiler with ActionScript target. Even further, within Alchemy, your code lives inside the reserved fast memory, and access to for instance malloc'ed pieces of memory are inside this memory.

Conclusion and final shout out to Adobe

This leaves ActionScript behind. Not because of the language, just because of the compilers. ActionScript doesn't support inlining, nor does the compiler do any optimizing steps. We could start modifying our own asc.jar and doing this, but with the Flex SDK being opensource, it's still not legal to modify the asc.jar and release, essentially making that step useless to do.

All in all, ActionScript isn't able to target it's own platform as well as both Haxe and Alchemy. Sure, you can insert optimized hand written bytecode in performance critical places, but at that point, why shouldn't I just switch to using Haxe or Alchemy ?  In my opinion, both have their downsides.

Alchemy is great, but as it's currently in beta, it's not even an industry standard. Sure, C as a language is, but Alchemy just isn't. Also, when using Alchemy, even the smallest piece of code results in a huge .swc because of all the standard C library code. All in all, it's amazing Alchemy runs this fast considering all the steps the code takes before it actually runs on your machine.

Haxe is a work of genius. It's a great compiler, I like the language and I specifically love the .abc format it contains, allowing me to write bytecode inside the Flash runtime and executing it. In my book, that's awesome. As of today, Haxe can even target the iPhone ! The language is great and has many features ActionScript lacks. It baffles me how one man army nicolas can single handedly write a language and a multi target compiler, and in many respects stay on top of the ActionScript compiler.

My first problem with Haxe is just that. The one man army. Should nicolas decide he doesn't feel like it anymore, the platform is dead. Sure, there's more people working on it, but in it's core it seems to be that nicolas is the driving force. My second problem is that although there is a huge pool of ActionScript developers, which can work together within the industry standard called ActionScript. There's not even near that many people who've even touched haxe, let alone the number of developers who are able to call themselves "Haxe Professionals".

Instead, I'll stick with ActionScript for now, and hand roll my optimized code using pure abc, haxe or Alchemy, which ever suits the purpose best. But the necessity to do so leaves me wondering what the state of ActionScript and it's compilers is. Should we start building our own ActionScript compiler ? Add our own features to the language ?

[EDIT]

What I failed to mention is that due to the overhead the asc.jar added to the bytecode (some return's and at every function a getlocal0 + pushscope I didn't need) I eventually switched from using that to moving to Haxe with it's Haxe format library. One of the supported formats is ABC, and I ended up using that to create my own .abc bytecode, injecting that instead of the asc.jar output. Again, I'm not saying either Haxe or Alchemy are bad tools, I'm just saying currently ActionScript is lagging behind.

No related posts.

After the overwhelming response of posting the sources of 300.000 pixels with Alchemy and Pixelbender, I thought it would be nice to post some more from my session at FITC Amsterdam.

First let's take a look at what people have done in response of the previous sources.

Yonatan Offek - Sierpinski Particles

Peter Nitsch - Alchemy Particles

David Lenaerts - Smoke, Milk and Ink

Dennis Ippel - Incredibly fast Plasma

Joa Ebert - Massive amounts of 3D particles without Alchemy and Pixelbender

All the work in that list is incredible, but for this post, I'm most interested in the last 2. Joa's work on getting AS3 up to speed with my Alchemy demo (Joa's demo is actually faster then the Alchemy version on my machine) is incredible. And it also proves the one thing I tried to convey in my blog post Adobe Alchemy, is it ActionScript Heresy ?.

Joa's optimization relies mostly on a linked list implementation to iterate over the particles, and using the Flash 10 Vector type to allow for faster access to write to screen. Another speed improvement over the Alchemy version is not using Pixelbender, but having both rotation and projection done in the same loop as drawing to screen. All in all, his result on this is pretty awesome.

Again, writing highly optimized ActionScript should enable you to do the same speedwise as Alchemy compiled code. There's at least one definitive advantage which Alchemy has in that case. The first advantage in Alchemy lies is the fast ByteArray access (which currently only Alchemy and Haxe can use).

The second advantage is why I think we should have a better AS3 compiler. It is the use of a more advanced compiler. This reduces the need to *highly* optimize your code, although I'd always recommend doing that in any case. My demo's here are meant of examples on how to use Alchemy, rather then to get the last bit of performance out. That, we'll leave to PapervisionX work.

Now, as I said, for this post we would be looking at two of the responses on my earlier Alchemy demo, the second one being Dennis Ippel's Fast Plasma effects. Plasma effects and Lookup Table efffects aren't too different. Ricardo Cabello did some experiments with them earlier, using Pixelbender. I worked on taking this type of effect and implementing it using Alchemy. Since that I had already done some experiments with them earlier, I thought it would be nice to release the sources, and see what you guys do with them. Maybe Joa feels like another shot of doing the same with ActionScript ?

Example

Source

Play around, let me know what you do with it, and make sure you check out Mr. Quilez his webpage, specifically, the article on Plane Deformations, where part of this code is derived from. If you want to optimize more, you *could* read Kebby's article F***ing learn to code again.

One more observation  : the difference between running this on Firefox on a Mac and just running it in the FlashPlayer is huge, even bigger then normal. Could someone from Adobe finally explain this ?

No related posts.

Pushing around +300.000 3D particles, realtime, on screen, using Flash ? No problem, if you are using Adobe Alchemy & PixelBender to compile and run your code!

During my session "professionally pushing pixels" at FITC Amsterdam this year, amongst other things, I talked about how to best utilize parts of the Flash Player to get top performance. This is one of the examples I showed. What you are seeing in this example, is +300.000 particles being 3D transformed, projected and draw to 2D. And it does so at quite a good framerate (well, it'll depend on your machine too).

So, how do we achieve this ? The answer is a combination of PixelBender and Alchemy.

First, let's look at the 3D transformation and projection.

Flash 10 has a number of native features to allow for 3D transformation and projection. You'll find that this is a combination of using the Vector, Vector3D, Matrix3D, PerspectiveProjection, etc. Although these features are great, we can't use them in combination with Alchemy easily. I'll explain why later, for now, let's look at an alternative method to do the projection.

Where oh where in the Flash Player do we have a method of doing very fast math ? The answer is; pixelbender! Although Pixelbender is normally used for image based manipulation, you can make it do any type of number-crunching which is able to be executed in parallel and without loops.

To calculate rotations and projecting our 3D data, we use Pixelbender in "ShaderJob" mode. When using pixelbender in image based mode, it operates in 8 bits per channel. Thankfully, when using it with a ShaderJob, it allows 32 bits precision per channel for the data processing. Since 8 bit precision wouldn't be enough for this example, we use a shaderjob.

The VertexProjector pixelbender kernel, included with the source is a simple way of transforming and projecting vertices (representing particles, in this case) in 3D space. We feed this kernel a bytearray of x,y,z paired data, and execute the shaderjob. It then returns the data as a bytearray, in px, py, pz format.

Drawing things to screen.

Now we have all the 2D projected 3D data, we need to draw things to screen, and we have to do so as quickly as possible. This step is traditionally called rasterization. In AS3, you're most likely to use getPixel when drawing on a per pixel basis. Doing so in a loop for 300.000 pixels turns out to be very slow. The solution for this would be to optimize that loop as much as possible. Either by writing your own bytecode, or maybe writing your own post-processor for you code, before you compile. But we don't have too, since Adobe Alchemy exists.

As you can read in my earlier post about Adobe Alchemy, I openly questioned why it was so speedy, as compared to regularly compiled ActionScript 3 code. Although the answer is rather complex, the combination of C Based code, the LLVM compiler and "Alchemy Virtual Memory" are the base of this. The large difference between Alchemy compiled actionscript and regular compiled Actionscript can be further explained by the regular AS3 compiler not doing any optimisation. This example shows off those performance increases.

One thing to worry about when using Alchemy in your ActionScript projects is marshalling. You can read Branden Hall's post on Alchemy for more info on that. Since we wouldn't be able to marshal 300.000 vertices from a Vector.<Number> in AS3 to our alchemy code, we need to find a better solution. This is exactly why we are using Pixelbender and more-over, the bytearray data.

It is possible to manipulate the memory Alchemy uses in the runtime. This memory is represented as an AS3 ByteArray object. If we directly write and get our data from this memory block, no marshalling is needed. Although this means not all things can be done this way, for some things, this can be very useful. For instance, getting large blocks of data, like images and bytearrays of coordinates.

Getting all these 3D particles to screen is simply 1 inner loop. While we would normally call setPixel for that, in Alchemy code, we don't have that luxury. Instead of that, we write directly to our screenbuffer memory, which is represented as a set of int's. Here, one more problem comes into play. Endianess, defines the byte ordering for a set of data. Alchemy uses little-endianess for it's internal memory representation. Specificall, it uses a small class called LEByteArray. This class extends ByteArray and ensures no changes are made to the endianess of the memory. Makes sense, since otherwise your code would blow up.

Writing to the screen is then a piece of cake. We take the alchemy processed data from it's memory, and write it to a bitmapdata using the formerly much less usable setPixels() command. It's amazing to see how fast this is.

Look at the example here, and download the full sourcecode here. As you can see from the example, the difference between doing this with regular ActionScript versus Alchemy nears a 5 fold speed increase.

Thanks to Keith Peters, for providing me with the 3D Strange Attractor code! And additional thanks to Mr.Doob for the stats object.

In future I'll be posting more demos of the technology. Amongst which there will be one appliance for the future version of Papervision3D, PapervisionX.

No related posts.