Planet Parrot

parrot.org: Parrot 5.3.0 "W00tstock Parrot" Released!

Util — 2013-04-18T20:36:07+00:00

We are stardust.
Billion year old carbon.
We are golden.
Caught in the devil's bargain
And we've got to get ourselves back to the garden.
(To some semblance of a garden.)
  -- "Woodstock", by Joni Mitchell

On behalf of the Parrot team, I'm proud to announce Parrot 5.3.0, also known as "W00tstock Parrot". Parrot is a virtual machine aimed at running all dynamic languages, and currently focusing on Perl 6.

parrot.org: The Parrot Foundation is accepted for GSOC 2013

rurban — 2013-04-12T17:47:20+00:00

dukeleto: Happy to announce that @parrotvm will be mentoring students in #gsoc again this year! If you know awesome CS students, send them to me :)

We need mentors to sign up at
http://www.google-melange.com/gsoc/org/google/gsoc2013/parrot
and project ideas to be collected/edited/reviewed at the "Ideas page".

Tadeusz Sośnierz (tadzik): Polish Perl Workshop status update

ttjjss — 2013-03-27T11:56:41+00:00

Exactly two months remain until Polish Perl Workshop this year. In case you didn’t know, I’m shamelessly reposting the status update from our ACT website.

We’ve got the first sponsor offers, and about 15 talk submissions so far. Awesome! However, as we are well aware that most of you will wait to submit talks until there’s less than 2 days until the deadline (or later :)) I’m hereby announcing: The talk submission deadline for Polish Perl Workshop 2013 is 14 of April. That’s about two weeks to finally make up your mind and submit something. Please do! There’ll be only one chance to talk at the First Polish Perl Workshop: there’ll be no other :)

Of course, if you’d prefer to organize more of a hands-on event, we have the entire sunday dedicated to hackathon, BoFs, classes etc. If you want to submit something more lenghty, that’s the right way to do it.

In other interesting news, we started looking at conference t-shirts (yes, they’re free for all attendees). We have something on our mind that never happened before on a Perl conference (or at least I’ve never seen or heard of something like this). We’ll keep you informed.

parrot.org: Parrot 5.2.0 "Stuffed Parrot" Released!

Util — 2013-03-23T17:59:00+00:00

I am not dead yet              I can dance and I can sing
I am not dead yet             I can do the Highland Fling
I am not dead yet                    No need to go to bed
No need to call the doctor         Cause I'm not yet dead
    -- Spamalot

On behalf of the Parrot team, I'm proud to announce Parrot 5.2.0,
also known as "Stuffed Parrot". Parrot
is a virtual machine aimed at running all dynamic languages.

parrot.org: Parrot 5.1.0 "Zombie Parrot" Released!

Util — 2013-02-20T06:39:13+00:00

    Flat on the bunk again, he ran for his life. The Parrot stalked him
    through the grey hours of morning, smoothing its fractal feathers,
    shuffling itself slowly into clarity as though at the end of a
    flashy film-dissolve, until at last his mind's eye had to acknowledge
    a shape,
        a shape,
            a wink
-- From BLIT, a short story by David Langford
    http://www.infinityplus.co.uk/stories/blit.htm

On behalf of the Parrot team, I'm proud to announce Parrot 5.1.0,
also known as "Zombie Parrot". Parrot
is a virtual machine aimed at running all dynamic languages.

cotto: It's Been Quiet

Christoph Otto — 2013-02-16T07:51:22+00:00

It's been quiet. Too quiet.

Interest in Parrot has waned over the past 18 months. The most recent flurry of activity happened when Allison Randal brought up the fact that The Parrot Foundation was in shambles and suggested shutting it down. This naturally brought up the state of Parrot itself and what the future holds for it, if anything. The situation is perhaps less than ideal. The short answer is that Parrot's immediate prospects are iffy at best, but there is at least one niche where Parrot still has a chance to shine.

The surface problem with Parrot is that there’s a lack of people who can find the tuits to hack on it these days. Different people have their own analyses as to why this is happening. My best answer is that Parrot doesn’t have a compelling value proposition. Hosting every dynamic language was pretty revolutionary around the time Parrot was started more than a decade ago. Today that’s no longer the case and the bigger language runtimes like the JVM, CLR and JavaScript (not a VM but a very poplar compilation target) can run circles around Parrot on most of the axes that matter.

Those of us who care about Parrot need to find a way to make it matter and to do so quickly.

Rakudo is the current most complete and active language implementation that runs on Parrot, and even *it* is moving toward running on many backends. Parrot’s best bet is to focus exclusively on supporting Rakudo and give it a reason to stick around. If supporting all dynamic languages was ever a good idea for Parrot, that’s no longer the case. The reality of Parrot’s effective niche has become much harder to ignore. The best move is to adapt accordingly.

Parrot has been inactive (among many reasons) because its developers can see that the goal of hosting all dynamic languages isn’t realistically attainable given Parrot's current resources. With a new and more tightly defined plan, Parrot has a fighting chance to find a useful niche.

Parrot's new niche and reason for existence needs to be to support Rakudo and nqp until those languages either fail, succeed, or have no further use for Parrot.

This will be a liberating shift for Parrot. The official policy is now “make nqp and Rakudo better”. Within that constraint, any change is welcome. In a bit more detail, the two goals by which any potential change should be judged are:

1) Does it provide a benefit to Rakudo, especially a *measurable* *non-theoretical* benefit?

If a change makes Rakudo happy, sold! This includes requested features, optimizations, bug fixes and the like. This is *the* primary concern and the best way to provide value to nqp and Rakudo.

2) Does it make Parrot’s code simpler without increasing complexity elsewhere?

Simplifying Parrot is valuable, but only in a much more indirect way. This goal is a distant second in importance to performance improvements. That said, simplifying Parrot is still helpful. Some of Parrot’s problems come from the decade of accumulated cruft. A simpler Parrot is more approachable and easier to profile, maintain and debug. Simplicity should be pursued as long as that simplicity doesn't mean shuffling complexity elsewhere and *especially* if the simplification comes with a performance bump.

That’s all there is to it. With simple and immediate rules rather than a slow and deliberate deprecation policy, half-done features that were kept around for years “just in case” can safely be removed.

Another implication of all this is that our deprecation and support policy are going away. They were well-intentioned but appropriate for a project in a much more mature and stable state. Our new support policy is “we’ll try to fix bugs and keep nqp running”. We’ll continue to make monthly releases but they will not be labelled as “supported” or “developer” as in the past.

Observers of Parrot will note by now that this isn’t the first time that Parrot has tried something radical. This isn’t even the first time that *I’ve* tried something radical. What's different this time is that we’re no longer trying to be all things to all languages; we’re trying to be one thing to one language that’s already our customer. This will still involve a ton of work, but the scope reduction shrinks the task from Herculean to merely daunting.

So here’s where you, the reader come in. Whether you’ve hacked on Parrot in the past or came for the lulz and accidentally got interested, you can help. The big goals are to make Parrot (and by extension nqp and Rakudo) smaller and faster. Below are a few specific ways you can help. Whatever you do though, don't make any changes that will be detrimental to nqp and Rakudo, and coordinate any backwards-incompatible changes before they get merged into Parrot master.

Grab a clone of Parrot and nqp. Build and install them. Play with the sixparrot branch, where some initial work is already in progress. Already there? Great! The next steps are a little harder.

Remove code paths that nqp doesn’t exercise. This can be single if statements or it can be whole sections of the source tree. Tests are the same as code; if the nqp and Rakudo’s tests don’t exercise them, out they go. Tests exist to increase inertia, but are only useful to the degree that they test useful features. When in doubt, either ask in #parrot or just rip it out and see what happens.

Relatedly, profile and optimize for nqp. If you like C, break out valgrind, build out a useful benchmark and see how fast you can make it run. If you find some code that doesn’t seem to be doing anything, you’ve just found an optimization!

Learn nqp and Perl 6. There’s been a lack of tribal knowledge about nqp’s inner workings ever since Parrot started distancing itself from Rakudo. We need to reverse that tendency so that nqp is regarded as an extension of Parrot.

Overall, the next few months will be interesting. I don't know if they'll result in success for Parrot, but I'm willing to give it one more shot.

parrot.org: Parrot 5.0.0 "Johnny Five Alive" Released!

dukeleto — 2013-01-23T04:24:29+00:00

"No disassemble!"
    -- Johnny Five

On behalf of the Parrot team, I'm proud to announce Parrot 5.0.0, also known
as "Johnny Five Alive". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.

Parrot 5.0.0 consists of 175 new commits from six authors on the master branch
since the 4.11.0 release. Notably, this is the first stable release of Parrot
with thread support (via the Task PMC).

Tadeusz Sośnierz (tadzik): Threads for Rakudo Perl 6

ttjjss — 2012-12-22T10:36:54+00:00

So it has come to this. Threads.pm is up and running, bringing the ever-wanted threaded execution to the most popular Perl 6 implementation.

You’re looking for TL;DR, aren’t you? Here’s what it’s capable of:

use Threads;
use Semaphore;

my @elements;
my $free-slots = Semaphore.new(value => 10);
my $full-slots = Semaphore.new(value => 0);

sub produce($id) {
    my $i = 0;
    loop {
        $free-slots.wait;
        @elements.push: $i;
        $i++;
        $full-slots.post;
    }
}

sub consume($id) {
    loop {
        $full-slots.wait;
        my $a = @elements.shift;
        $free-slots.post;
    }
}

for 1..5 -> $i {
    async sub { produce($i)  }
}

for 5..10 -> $i {
    async sub { consume($i) }
}

Doesn’t look that awesome. I mean, it’s just a producer-consumer problem, what’s the big deal? Let me repeat:

OMG, RAKUDO HAS WORKING THREADS.

So, once we’re done celebrating and dancing macarena all around, there’ll always be someone to ask “hold on, there’s gotta be a caveat. Something surely is missing, explain yourself!”

I’ll be delighted to say “nope, everything’s there!”, but that’d make me a liar. Yeah, there are missing pieces. First, those aren’t really native threads – just green threads. Native OS threads are already implemented in Parrot VM, but NQP (the language that Rakudo is based on) still doesn’t support them, so before some volunteer comes along to fix them, you’ll still have to build parrot --without-threads (which means: use green threads, not OS threads) for Threads.pm to work. But fear not! The API is exactly the same, so once native threads are there, both Threads.pm and the code you write with it should work without any changes.

But green threads are fine too! Except for one minor detail: whenever any of them blocks on IO, the entire Parrot comes to a halt. The plan is for Parrot threads scheduler to handle it nicely, but it’s not there yet, so if you expected nice and easy async IO, sorry, but you’re stuck on MuEvent :)

Yep, we’re not really there yet. But I claim it’s closer than ever. We have working threads implementation. You can write code with that, and it’s not a PITA. Go for it! There’s a lot of room to improve it. I didn’t try really hard for Threads.pm to follow the concurrency synopsis (in my defense, I think niecza doesn’t follow it either :)), and I think that once we unleash a wolfpack of developers which can work towards something that we’ll all love to use.

parrot.org: Parrot threads on the perl6 advent calendar - Day 11

rurban — 2012-12-18T22:24:15+00:00

Parrot threads were featured on the perl advent calendar, day 11. Something cool about Perl 6 every day, in December.

See http://perl6advent.wordpress.com/2012/12/11/day-11-parrot-threads/.

parrot.org: Parrot 4.11.0 "All together - Happy Birthday Lovebird" Released!

rurban — 2012-12-18T22:09:12+00:00

On behalf of the Parrot team, I'm proud to announce Parrot 4.11.0, also known as "All together - Happy Birthday Lovebird". Parrot is a virtual machine aimed at running all dynamic languages.

parrot.org: Parrot string encodings on the perl6 advent calendar - Day 7

rurban — 2012-12-08T15:07:02+00:00

Debugging parrot strings were featured on the perl6 advent calendar, day 7. Something cool about Perl 6 every day, in December.

See http://perl6advent.wordpress.com/2012/12/07/day-7-mimebase64-on-encoded-strings/.

Shows how to debug into crazy string encoding problems, when you are not sure if the core implementation, the library, the spec or the tests are wrong. It turned out, that the library and the tests were wrong.

parrot.org: Parrot 4.10.0 "Red-eared Parakeet" Released!

rurban — 2012-11-21T22:14:04+00:00

On behalf of the Parrot team, I'm proud to announce Parrot 4.10.0, also known as "Red-eared Parakeet". Parrot is a virtual machine aimed at running all dynamic languages.

Andrew Whitworth: More IO Work?

2012-11-21T08:00:00+00:00

I might not be too bright. Either that or I might not have a great memory, or maybe I’m just a glutton for punishment. Remember the big IO system rewrite I completed only a few weeks ago? Remember how much of a huge hassle that turned into and how burnt-out I got because of it? Apparently I don’t because I’m back at it again.

Parrot hacker brrt came to me with a problem: After the io_cleanup merge he noticed that his mod_parrot project doesn’t build and pass tests anymore. This was sort of expected, he was relying on lots of specialized IO functionality and I broke a lot of specialized IO functionality. Mea culpa. I had a few potential fixes in mind, so I tossed around a few ideas with brrt, put together a few small branches and think I’ve got the solution.

The problem, in a nutshell is this: In mod_parrot brrt was using a custom Winxed object as an IO handle. By hijacking the standard input and output handles he could convert requests on those handles into NCI calls to Apache and all would just work as expected. However with the IO system rewrite, IO API calls no longer redirect to method calls. Instead, they are dispatched to new IO VTABLE function calls which handle the logic for individual types.

First question: How do we recreate brrt’s custom functionality, by allowing custom bytecode-level methods to implement core IO functionality for custom user types?

My Answer: We add a new IO VTABLE, for “User” objects, which can redirect low-level requests to PMC method calls.

Second Question: Okay, so how do we associate thisnew User IO VTABLE with custom objects? Currently the get_pointer_keyed_int VTABLE is used to get access to the handle’s IO_VTABLE* structure, but bytecode-level objects cannot use get_pointer_keyed_int.

My Answer: For most IO-related PMC types, the kind of IO_VTABLE* to use is staticly associated with that type. Socket PMCs always use the Socket IO VTABLE. StringHandle PMCs always use the StringHandle IO VTABLE, etc. So, we can use a simple map to associate PMC types with specific IO VTABLEs. Any PMC type not in this map can default to the User IO VTABLE, making everything “just work”.

Third Question: Hold your horses, what do you mean “most” IO-related PMC types have a static IO VTABLE? Which ones don’t and how do we fix it?

My Answer: The big problem is the FileHandle PMC. Due to some legacy issues the FileHandle PMC has two modes of operation: normal File IO and Pipe IO. I guess these two ideas were conflated together long ago because internally the details are kind of similar: Both files and pipes use file descriptors at the OS level, and many of the library calls to use them are the same, so it makes sense not to duplicate a lot of code. However, there are some nonsensical issues that arise because Pipes and files are not the same: Files don’t have a notion of a “process ID” or an “exit status”. Pipes don’t have a notion of a “file position” and cannot do methods like seek or tell. Parrot uses the "p" mode specifier to tell a FileHandle to be in Pipe mode, which causes the IO system to select a between either the File or the Pipe IO VTABLE for each call. Instead of this terrible system, I suggest we separate out this logic into two PMC types: FileHandle (which, as it’s name suggests, operates on Files) and Pipe. By breaking up this one type into two, we can statically map individual IO VTABLEs to individual PMC types, and the system just works.

Fourth Question: Once we have these maps in place, how do we do IO with user-defined objects?

My Answer: The User IO VTABLE will redirect low-level IO requests into method calls on these PMCs. I’ll break IO_BUFFER* pointers out into a new PMC type of their own (IOBuffer) and users will be able to access and manipulate these things from any level. We’ll attach buffers to arbitrary PMCs using named properties, which means we can attach buffers to any PMC that needs them.

So that’s my chain of thought on how to solve this problem. I’ve put together three branches to start working on this issue, but I don’t want to get too involved in this code until I get some buy-in from other developers. The FileHandle/Pipe change is going to break some existing code, so I want to make sure we’re cool with this idea before we make breaking changes and need to patch things like NQP and Rakudo. Here are the three branches I’ve started for this:

whiteknight/pipe_pmc: This branch creates the new Pipe PMC type, separate from FileHandle. This is the breaking change that we need to make up front.
whiteknight/io_vtable_lookup: This branch adds the new IOBuffer PMC type, implements the new IO VTABLE map, and implements the new properties-based logic for attaching buffers to PMCs.
whiteknight/io_userhandle: This branch implements the new User IO VTABLE, which redirects IO requests to methods on PMC objects.

Like I said, these are all very rough drafts so far. All these three branches build, but they don’t necessarily pass all tests or look very pretty. If people like what I’m doing and agree it’s a good direction to go in, I’ll continue work in earnest and see where it takes us.

parrot.org: Parrot 4.9.0 "Proto-Hydra" Released!

dukeleto — 2012-10-27T22:47:25+00:00

"All proofs inevitably lead to propositions which have no proof!
All things are known because we want to believe in them."

   -- The Lady Jessica, to Bene Gesserit delegation

On behalf of the Parrot team, I'm proud to announce Parrot 4.9.0, also known as "Proto-Hydra". Parrot is a virtual machine aimed at running all dynamic languages.

parrot.org: Parrot 4.8.0 "Spix's Macaw" Released!

ayardley — 2012-09-19T04:08:33+00:00

On behalf of the Parrot team, I'm proud to announce Parrot 4.8.0,
also known as "Spix's Macaw". Parrot
is a virtual machine aimed at running all dynamic languages.

Andrew Whitworth: September Status

2012-09-14T07:00:00+00:00

First, some personal status:

Personal Status

I haven’t blogged in a little while, and there’s a few reasons for that. I’ll list them quickly:

Work has been…tedious lately and when I come home I find that I want to spend much less time looking at a computer, especially any computer that brings more stress into my life. Also,
My computer at home generates a huge amount of stress. In addition to several physical problems with it, and the fact that I effectively do not have a working mouse (the built-in trackpad is extremely faulty, and the external USB mouse I had been using is now broken and the computer won’t even book if it’s plugged into the port), I’ve been having some software problems with lightdm and xserver crashing and needing to be restarted much more frequently than I think should be needed. We are planning to buy me a new one, but the budget won’t allow that until closer to xmas.
The io_cleanup1 work took much longer than I had anticipated. I wrote a lot more posts about that branch than I ever published, and the ones I did publish were extremely repetitive (“It’s almost finished, any day now!”). Posting less means I got out of the habit of posting, which is a hard habit to be in and does require some effort.

I’m going to do what I can to post something of a general Parrot update here, and hopefully I can get back in the habit of posting a little bit more regularly again.

`io_cleanup1` Status

io_cleanup1 did indeed merge with almost no problems reported at all. I’m very happy about that work, and am looking forward to pushing the IO subsystem to the next level. Before I started io_cleanup1, I had some plans in mind for new features and capabilities I wanted to add to the VM. However, I quickly realized that the house had some structural problems to deal with before I could slap a new coat of paint on the walls. The structure is, I now believe, much better. I’ve still got that paint in the closet and eventually I’m going to throw it on the walls.

The io_cleanup branch did take a lot of time and energy, much more than I initially expected. But, it’s over now and I’m happy with the results so now I can start looking on to the next project on my list.

Threads Status

Threads is very very close to being mergable. I’ve said that before and I’m sure I’ll have occasion to say it again. However there’s one remaining problem pointed out by tadzik, and if my diagnosis is correct it’s a doozie.

The basic threads system, which I outlined in a series of blog posts ages ago goes like this: We cut out the need to have (most) locks, and therefore we cut out many possibilities of deadlock, by making objects writable only from the thread that owns them. Other threads can have nearly unfettered read access, but writes require sending a message to the owner thread to perform the update in a synchronized, orderly manner. By limiting cross-thread writes, we cut out many expensive mechanisms that would need to be used for writing data, like Software Transactional Memory (STM) and locks (and, therefore, associated deadlocks). It’s a system inspired closely by things like Erlang and some functional languages, although I’m not sure there’s any real prior art for the specifics of it. Maybe that’s because other people know it won’t work right. The only thing we can do is see how it works.

The way nine implemented this system is to setup a Proxy type which intercepts and dispatches read/write requests as appropriate. When we pass a PMC from one thread to another, we instead create and pass a Proxy to it. Every read on the proxy redirects immediately to a read on the original target PMC. Every write causes a task to dispatch to the owner thread of the target PMC with update logic.

Here’s some example code, adapted from the example tadzik had, which fails on the threads branch:

function main[main](var args) {
    var x = 1;
    var t = new 'Task'(function() { x++; say(x); });
    ${ schedule t };
    ${ wait t };
    say("Done!");
}

Running this code on the threads branch creates anything from an assertion failure to a segfault. Why?

This example creates a closure and schedules that closure as a task. The task scheduler assigns that task to the next open thread in the pool. Since it’s dispatching the Task on a new thread, all the data is proxied. Instead of passing a reference to Integer PMC x, we’re passing a Proxy PMC, which points to x. This part works as expected.

When we invoke a closure, we update the context to point to the “outer” context, so that lexical variables (”x”, in this case) can be looked up correctly. However, instead of having an outer which is a CallContext PMC, we have a Proxy to a CallContext.

An overarching problem with CallContext is that they get used, a lot. Every single register access, and almost all opcodes access at least one register, goes through the CallContext. Lexical information is looked up through the CallContext. Backtrace information is looked up in the CallContext. A few other things are looked up there as well. In short, CallContexts are accessed quite a lot.

Because they are accessed so much, CallContexts ARE NOT dealt with through the normal VTABLE mechanism. Adding in an indirect function call for every single register access would be a huge performance burden. So, instead of doing that, we poke into the data directly and use the raw data pointers to get (and to cache) the things we need.

And there’s the rub. For performance we need to be able to poke into a CallContext directly, but for threads we need to pass a Proxy instead of a CallContext. And the pointers for Proxy are not the same as the pointers for CallContext. See the problem?

I identified this issue earlier in the week and have been thinking it over for a few days. I’m not sure I’ve found a workable solution yet. At least, I haven’t found a solution that wouldn’t impose some limitations on semantics.

For instance, in the code example above, the implicit expectation is that the x variable lives on the main thread, but is updated on the second thread. And those updates should be reflected back on main after the wait opcode.

The solution I think I have is to create a new dummy CallContext that would pass requests off to the Proxied LexPad. I’m not sure about some of the individual details, but overall I think this solution should solve our biggest problem. I’ll probably play with that this weekend and see if I can finally get this branch ready to merge.

Other Status

rurban has been doing some great cleanup work with native PBC, something that he’s been working on (and fighting to work on) for a long time. I’d really love to see more work done in this area in the future, because there are so many more opportunities for compatibility and interoperability at the bytecode level that we aren’t exploiting yet.

Things have otherwise been a little bit slow lately, but between io_cleanup1, threads and rurban’s pbc work, we’re still making some pretty decent progress on some pretty important areas. If we can get threads fixed and merged soon, I’ll be on to the next project in the list.

Patrick R. Michaud: A Rakudo Performance

pmichaud — 2012-09-02T23:00:26+00:00

At YAPC::NA 2012 in Madison, WI I gave a lightning talk about basic improvements in Rakudo’s performance over the past couple of years. Earlier today the video of the lightning talks session appeared on YouTube; I’ve clipped out my talk from the session into a separate video below. Enjoy!

Andrew Whitworth: io_cleanup1 Lands!

2012-08-27T07:00:00+00:00

FINALLY! The big day has come. I’ve just merged whiteknight/io_cleanup1 to master. Let us rejoice!

When I started the project, months ago, I had intended to work on the branch for maybe a week or two at the most. Get in, clean what I could, get out. Wash, rinse, repeat. That’s exactly why I named the branch “io_cleanup1”, because I intended it to just be the first of what would be a large series of small branches. Unfortunately as I started cleaning I was lead to other things that needed to go. And those things lead elsewhere. Before I new it I had deleted just about all the code in all the files in src/io/* and started rewriting from the ground up.

Sometimes sticking with a plan and breaking up projects into small milestones is a good thing. Othertimes when you know what the final goal is and you’re willing to put in the effort, it’s good to just go there directly. That’s what I ended up doing.

To give you an idea of what my schedule was originally, I had intended to get this first branch wrapped up and merged before GSOC started, so that I could keep my promise of implementing 6model concurrently with that program. With GSOC over last week (I’ll write a post-mortem blog entry about it soon), I’ve clearly failed at that. I’m extremely happy with the results so far and given the choice I would not go back and do things any differently. The IO system was in terrible condition and it desperately needed this overhaul. I wish it hadn’t taken me so long, but with a system that’s so central and important, it was worthwhile taking the extra time to make sure things were correct.

Where to go from here? My TODO list for the near future is very short:

Threads
6model
More IO work

The Threads branch, the magnum opus of Parrot hacker nine is 99.9% of the way there. If we can just push it up over the cliff, we should be able to merge soon and open up a whole new world of functionality and cool features for Parrot. I’m already planning out all the cool additions to Rosella I’m going to make once threads are merged: Parallel test harness. Asynchronous network requests, an IRC client library. The addition of a real, sane threading system opens up so many avenues to us that really haven’t been available before. Sure there are going to be plenty of hiccups and speedbumps to deal with as we really get down and start to use this system for real things, but the merge of the threads branch represents a huge step forward and a great foundation to build upon.

I’m going to be putting forward as much effort as I can to getting this branch wrapped up and merged. Some of the remaining problems only manifest on hard-to-test platforms, which is where things start to get tricky. As I mentioned in an email to parrot-dev a while ago, test reports on rare platforms are great, but if we can’t take action on the reported failures we can get ourselves into something of a bind. The capability to find problems on those platforms and the capability to fix problems on those platforms are two very different capabilities. But, most of the time that’s a small issue and we’re going to just have to find a way to muscle through and get this branch merged one way or the other. If we can merge it without purposefully excluding any platforms, that would be great.

Before anybody thinks that I’m done with IO and that system is now complete, think again. There is still plenty of work to be done on the IO subsystem, and all sorts of cool new features that become possible with the new architecture and unified type semantics. I want to separate out Pipe logic from FileHandle into a new dedicated PMC type. Opening FileHandles in “p” mode for pipes is clumsy at best, and I want a more sane system. And while I’m at it, 2-way and 3-way pipes would make for a great feature addition (we can’t currently do these in any reliable way).

The one thing that has changed most dramatically in the new IO system is buffers. The buffering subsystem has not only been rewritten but completely redesigned. Instead of being type-specific they are now unified and type independent. Buffers are their own struct with their own API. Instead of having a single buffer that is used for both read and write, handles now have separate read and write buffers that can be created and managed independently. I want to create a new PMC type to wrap these buffers and give the necessary management interface so they can be used effectively from the PIR level and above.

Finally, the whiteknight/io_cleanup1 branch tried to stay as backwards compatible as possible, so many breaking changes I wanted to make had to wait until later. In the future expect to see many smaller branches to remove old broken features, old crufty interfaces, and old bad semantics. We’ll make these kinds of disruptive changes in much smaller batches, with more space between them.

parrot.org: Parrot 4.7.0 "Hispaniolan" Released!

Whiteknight — 2012-08-22T23:02:48+00:00

On behalf of the Parrot team, I'm proud to announce Parrot 4.7.0,
also known as "Hispaniolan". Parrot
is a virtual machine aimed at running all dynamic languages.

Andrew Whitworth: Parrot 4.7.0 "Hispaniolan" Released!

2012-08-22T07:00:00+00:00

On behalf of the Parrot team, I’m proud to announce Parrot 4.7.0, also known as “Hispaniolan”. Parrot is a virtual machine aimed at running all dynamic languages.

Parrot 4.7.0 is available on Parrot’s FTP site, or by following the download instructions at http://parrot.org/download. For those who would like to develop on Parrot, or help develop Parrot itself, we recommend using Git to retrieve the source code to get the latest and best Parrot code.

Parrot 4.7.0 News:

- Core
    + Added .all_tags() and .all_tagged_pmcs() methods to PackfileView PMC
    + Several build and coding standards fixes

The SHA256 message digests for the downloadable tarballs are:

4360ac3dffafffaa00bce561c1329df8ad134019f76930cf24e7a875a4422a90 parrot-4.7.0.tar.bz2
c0bffd371dea653b9881ab2cc9ae5a57dc9f531dfcda0a604ea693c9d2165619 parrot-4.7.0.tar.gz

Many thanks to all our contributors for making this possible, and our sponsors for supporting this project. Our next scheduled release is 18 September 2012.

The release is indeed out a day late. It’s not that I forgot about it, it’s just that I can’t read a calendar and HOLY CRAP, IT’S WEDNESDAY ALREADY? When did that happen? So, and I can’t stress this enough, Mea Culpa.

parrot.org: Results

brrt — 2012-08-19T22:05:47+00:00

Tomorrow is google's appointed 'firm pencils down' date. That seems like a good time to discuss the results of my work on mod_parrot so far.

mod_parrot is, as I have mentioned before, a two-layered system, with one half interfacing with apache (the module) and the other half with the interpreter and the compilers, the 'loader'. There is also a vital third component, the test system called pudding.

parrot.org: A new hope

brrt — 2012-08-11T17:40:54+00:00

This week I finally got arround to giving a new, fresh structure to the mod_parrot module code. I have complained, perhaps not loudly enough, about the various inadequacies of the old codebase, mostly with regards to extensibility. A cleanup was needed. As such, here is a walkthrough of the new structure, also serving as documentation.

parrot.org: Interpreters with butterfly wings

brrt — 2012-08-03T08:58:01+00:00

I for one am totally for whimsical blog titles, wouldn't you agree? In other news, after a lull of two weeks (codewise at least) I've finally started to work on mod_parrot again. The big (dis)?advantage from not working on code is that you start to think more of what you could do (and should have done), rather than what you have done.

As it turns out, I handle interpreters in a rather confused manner. My goal for the next two weeks is to fix that. First, let me describe what should be done to correctly run a script on an interpreter using mod_parrot:

parrot.org: PACT: Adjusting the schedule

benabik — 2012-07-24T18:43:00+00:00

I appear to be continuing my weekly blogging every 14 days. Ah, well. My progress has been fairly intermittent as I work out this whole "getting sleep with a newborn around", but I'm starting to make real progress again. Today's blog will discuss what I've done in the last couple weeks and an updated schedule for the next month.

My progress can be split into a few topics: syntax highlighting, style changes, bug fixes, test helpers, and tests themselves.

Andrew Whitworth: io_cleanup1 Done?

2012-07-22T07:00:00+00:00

This morning I made a few last commits on my whiteknight/io_cleanup1 branch, and I’m cautiously optimistic that the branch is now ready to merge. The last remaining issue, which has taken the last few days to resolve, has been fixing readine semantics to match some old behavior.

A few days ago I wrote a post about how complicated readline is. At the time, I thought I had the whole issue under control. But then Moritz pointed out a problem with a particular feature unique to Socket that was missing in the new branch.

In master, you could pass in a custom delimiter sequence as a string to the .readline() method. Rakudo was using this feature like this:

str = s.readline("\r\n")

Of course, as I’ve pointed out in the post about readline and elsewhere, there was no consistency between the three major builtin types: FileHandle, Socket and StringHandle. The closest thing we could do with FileHandle is this:

f.record_separator("\n");
str = f.readline();

Notice two big differences between FileHandle and Socket here: First, FileHandle has a separate record_separator method that must be called separately, and the record separator is stored as state on the FileHandle between .readline() calls. Second, FileHandle’s record separator sequence may only be a single character. Internally, it’s stored as an INTVAL for a single codepoint instead of as a STRING*, even though the .record_separator() method takes a STRING* argument (and extracts the first codepoint from it).

Initially in the io_cleanup1 branch I used the FileHandle semantics to unify the code because I wasn’t aware that Socket didn’t have the same restrictions that FileHandle did, even if the interface was a little bit different. I also didn’t think that the Socket version would be so much more flexible despite the much smaller size of the code to implement it. In short, I really just didn’t look at it closely enough and assumed the two were more similar than they actually were. Why would I ever assume that this subsystem ever had “consistency” as a driving design motivation?

So I rewrote readline. From scratch.

The new system follows the more flexible Socket semantics for all types. Now you can use almost any arbitrary string as the record separator for .readline() on FileHandle, StringHandle and Socket. In the whiteknight/io_cleanup1 branch, as of this morning, you can now do this:

var f = new 'FileHandle';
f.open('foo.txt', 'r');
f.record_separator("TEST");
string s = f.readline();

…And you can also do this, which is functionally equivalent:

var f = new 'FileHandle';
f.open('foo.txt', 'r');
string s = f.readline("TEST");

The same two code snippets should work the same for all built-in handle types. For all types, if you don’t specify a record separator by either method, it defaults to “\n”.

Above I mentioned that almost any arbitrary string should work. I use the word “almost” because there are some restrictions. First and foremost, the delimiter string cannot be larger than half the size of the buffer. Since buffers are sized in bytes, this is a byte-length restriction, not a character-length restriction. In practice we know that delimiters are typically things like “\n”, “\r\n”, ”,”, etc. So if the buffer is a few kilobytes this isn’t a meaningful limitation. Also, the delimiter must be the same encoding as the handle uses, or it must be able to convert to that encoding. So if your handle uses ascii, but you pass in a delimiter which is utf16, you may see some exceptions raised.

I think that the work on this branch, save for a few small tweaks, is done. I’ve done some testing myself and have asked for help to get it tested by a wider audience. Hopefully we can get this branch merged this month, if no other problems are found.

parrot.org: 4.6.0 Parrot "Wild Parrots of Telegraph Hill" Released

rurban — 2012-07-17T15:45:46+00:00

"Wild Parrots of Telegraph Hill"

parrot.org: As long as hope remains

brrt — 2012-07-16T12:17:45+00:00

So, this was rather an unproductive week, unfortunately. I'm completely busy with moving right now (and will be coming week). What did happen is that I poked a hole into parrot, and the community (nine) fixed it. The story: I started my 'loader' script by directly invoking a subroutine. That by-passed the starting of a green thread on the interpreter, which caused a crash when I tried to do something with that thread, such as sleeping. Nine fixed this issue by starting a green thread upon invocation using the api, which causes my tests to crash no more.

parrot.org: PACT: Spinning of the Wheels

benabik — 2012-07-12T19:58:14+00:00

So my 'vacation' was a visit to the hospital for the birth of my son. Now that this has happened, my schedule is going to be even more fun. Was in the hospital for most of a week and am now adjusting to life back home. I've been slowly turning my disassembler program into a "library" of sorts so I can call it repeatedly from tests.

Now to write some tests that convert PIR to Packfiles and Packfiles to PACT.Packfiles...

Andrew Whitworth: IO Cleanups Home Stretch

2012-07-10T07:00:00+00:00

I had made a round of fixes with regards to encodings in the whiteknight/io_cleanup1 branch a few days ago. Rakudo hacker Moritz was able to take a look at Rakudo’s spectests and verify that more tests were indeed passing because of it. The remaining test failures represent the changing semantics for the read method and what appear to be two genuine regressions or bugs.

Hopefully I will be able to get all these things sorted out this week before I go away on a mini vacation next weekend. Otherwise I can’t imagine this branch gets merged before the 4.6 release this month.

A few days ago I wrote a post about readline and some of the intricacies involved in that, and some of the weird semantics that I was attempting to unify. It turns out that some of these semantics are a major cause in one of the last bugs in the branch. Let’s look at some code in master to see where the hangup is. First, readline on a Socket:

METHOD readline(STRING *delimiter    :optional,
                INTVAL has_delimiter :opt_flag) {
    INTVAL idx;
    STRING *result;
    STRING *buf;
    GET_ATTR_buf(INTERP, SELF, buf);

    if (!has_delimiter)
        delimiter = CONST_STRING(INTERP, "\n");

    if (Parrot_io_socket_is_closed(INTERP, SELF))
        RETURN(STRING * STRINGNULL);

    if (buf == STRINGNULL)
        buf = Parrot_io_reads(INTERP, SELF, CHUNK_SIZE);

    while ((idx = Parrot_str_find_index(INTERP, buf, delimiter, 0)) < 0) {
        STRING * const more = Parrot_io_reads(INTERP, SELF, CHUNK_SIZE);
        if (Parrot_str_length(INTERP, more) == 0) {
            SET_ATTR_buf(INTERP, SELF, STRINGNULL);
            RETURN(STRING *buf);
        }
        buf = Parrot_str_concat(INTERP, buf, more);
    }

    idx += Parrot_str_length(INTERP, delimiter);
    result = Parrot_str_substr(INTERP, buf, 0, idx);
    buf = Parrot_str_substr(INTERP, buf, idx, Parrot_str_length(INTERP, buf) - idx);
    SET_ATTR_buf(INTERP, SELF, buf);
    RETURN(STRING *result);
}

We can ignore the fact that this implementation of readline doesn’t call Parrot_io_readline like every other PMC does. Or that if we did call that function the program would throw an exception because Parrot_io_readline doesn’t support sockets anyway. Whatever. Moving on…

For comparison, let’s look at the version from the Handle PMC (which is inherited by FileHandle):

METHOD readline() {
    STRING * const string_result = Parrot_io_readline(INTERP, SELF);
    RETURN(STRING *string_result);
}

The Socket version takes a delimiter parameter which is a STRING. When doing readline on a Socket, you can pass in any arbitrary string which is used as the token for end of line. With FileHandle, you don’t seem to have that. However, you can definitely use custom delimiters with FileHandle. However, we clearly don’t take a delimiter here and we aren’t passing one in as an argument to Parrot_io_readline like we do in the branch. Let’s see how it’s done instead. Here’s a snippet from Handle PMC:

    ATTR INTVAL    record_separator;  /* Record separator (only single char supported) */

We don’t need to look at any other code. This is the smoking gun. Socket.readline() can take any arbitrary STRING to use as a record separator, but FileHandle.readline() can only use a single codepoint, which it doesn’t take as an argument.

So that’s the problem right there. When I standardized the readline mechanics between types, I picked the FileHandle semantics. This was probably the wrong decision, because not only could Sockets use a more general mechanism but Rakudo relies on that behavior in its spectests. This does raise a question about why nobody ever expected this same behavior from FileHandle, or why the difference was not considered some kind of bug. It really goes to show how immature our IO system has been for all these years, and how we had all just grown accustomed to the arbitrary, inconsistent, nonsensical behaviors. It just works for some basic usages, so nobody ever complains about it. That time is, thankfully, coming quickly to an end.

Fixing this issue is actually going to take some serious work. Several function signatures are going to need updating to take a STRING delimiter instead of an INTVAL codepoint, and a major chunk of buffering logic is going to need to be rewritten to work on substrings instead of on individual codepoints. This, in turn, is going to require a heck of a lot more testing.

Last night I started putting in some of the changes necessary to use a substring terminator instead of a single codepoint. Most of what I’ve already done has been modifying function signatures. The real changes need to occur deep within the buffering logic and will require a little bit more time.

I’m looking forward to getting this branch fixed up and merged back to master so I can get to work on my next project. I think 6model is going to be the next thing I dig into, before I find something else that annoys me enough to put in a huge amount of effort to rewrite it. I’ll post more updates about my future projects and plans as I go.

parrot.org: Security API Update.

jharper1 — 2012-07-08T20:57:33+00:00

Working on flags and permissions this past weekend in security api. Slow and steady progress. In terms of the timeline I am behind, but I am making every effort to get back on track. Monday should be an interesting day to show this past weekends progress.

parrot.org: Pandora's Box

brrt — 2012-07-07T07:53:59+00:00

Or, the internal Parrot C API. It is open, now. At least, parts of it anyway, and hopefully somewhat limited in scope.

When I set out to write mod_parrot it was my goal to use the 'new' embedding API - the one with all the Parrot_api_* calls. This is a limited API designed for loading and running the parrot interpreter and some scripts. It isn't perfect or even elegant but it works. Moreover, People have Promised it to be Stable. However, because it was designed to be used outside of the parrot runloop, these functions are not re-entrant in a rather subtle manner.

parrot.org: Progress till now

Jashwanth — 2012-07-02T17:05:53+00:00

Till now was able to add the function to compute the eigenvalues. Also fixed the segmentation fault in the inverse function to do this has to edit the LU decomposition function to get results of couple of arrays. Now will be starting to work on the implementation of the function for eigenvectors and also try to develop the tests for inverse and eigenvalues functions.

Andrew Whitworth: HTML with Rosella Xml and Net

2012-06-30T07:00:00+00:00

HTML is a derivative of SGML, just like XML is. Sure, they look pretty much the same for the most part, but there are a few key differences that prevent HTML from being parsed exactly like XML. Part of the reason why I like XHTML so much is that it’s more usable with more parsers, including many of simpler and full-featured XML parsers. Simplicity in parsing was one of the original motivations of the XML design, at least in comparison to a full SGML parser or even something like a full HTML parser.

But that’s all besides the point.

I’ve been in something of a backyard gardening kick lately. We bought our house only a few short months ago, and are only half way through the first summer growing season in my modest little garden. My plans for next year are much more expansive. I’ve finally talked my wife into letting me buy some cherry trees to plant. She was also pretty willing to get a few grape vines planted (especially when I sketched out the beautiful wooden arbor they would be growing on). She put her foot down when I started talking about blueberries, apples and pears, however. And another garden bed or two for more vegetables. For some reason she’s convinced that we need some measure of open space in our little plot so the kid has somewhere to run and play. Some people have weird priorities.

This is all sort of besides the point too.

Getting the things I need for all this gardening work I’ve talked myself into is not cheap. Cherry tree seeds actually do grow on trees so that’s not a big deal, but other things like fertilizers, soil amendments, tools, materials for building a grape trellis and raised garden beds, not to mention a longer hose to reach all the new things that are going to require regular watering all cost money. And maybe a sprinler, like one of those fancy ones on an electronic timer. I can avoid some of that cost by getting things used and at discount on sites like Craigslist. So I’ve been going there. Every day.

And it’s tedious. I have to sort through hundreds of listings for things I don’t want, in categories that seem far too course. Sometimes, because things often get incorrectly categorized, I have to look in other related categories too, sorting through things that are even less relevant on average to try and find the occasional gem. This is all on top of the hardware-related problems I have being unable to use the trackpad on my laptop so web navigation on sites without keyboard shortcuts is an extreme pain. I start to think to myself: I can do better, I’m a programmer! For some values of “better” and “programmer”.

Enter Rosella. Now with Parrot, Winxed and Rosella I can use the Net library to fetch the text of the HTML code of the page. After some hacking in the last few days, I can parse that code with my Xml library (set in a new lenient mode) and start to work with it in a meaningful way:

function main[main]() {
    var rosella = load_packfile("rosella/core.pbc");
    Rosella.initialize_rosella("xml", "net", "string");

    var ua = new Rosella.Net.UserAgent.SimpleHttp();
    var response = ua.get("http://philadelphia.craigslist.org/w4m/");
    var doc = Rosella.Xml.read_string(response.content, false);

    doc.get_document_root()
        .get_children_named("body")
        .get_children_named("blockquote")
        .get_children_named("p", "row":[named("class")])
        .map(function(node) {
            return {
                "title": node.first_child("a").get_inner_xml(),
                "link":  node.first_child("a").attributes["href"],
                "price": node.first_child("span", "itempp":[named("class")]).get_inner_xml(),
                "has_pic": !Rosella.String.null_or_empty(
                    node.first_child("span", "itempx":[named("class")]).get_inner_xml()
                )
            };
        })
        .filter(function(obj) {
            return indexof(obj["title"], "compost") >= 0;
        })
        .map(function(obj) {
            return Rosella.String.format_obj("<a href='{link}'>{title} for {price}</a>", obj);
        })
        .foreach(function(string s) { say(s); });
}

That second argument to Rosella.Xml.read_string tells the parser to go into “non-strict” mode, which is basically my attempt to fudge the XML parsing rules to allow for the SGML nonsense in HTML. Without that, the parser will blow up pretty early in the parse because of unbalanced tags. The XML parser by default does not handle tags which are not balanced and which do not have the trailing slash to indicate a standalone tag, and the Craigslist source is filled with those kinds of things.

All I need to do is set this scraper up on a timer, and have it send me results somehow. If I set up a small server with mod_parrot and some kind of tool for generating RSS feeds, I could have this output neatly delivered to me on a regular basis. Considering that mod_parrot is moving along so smoothly and RSS is just another XML format, I think this is a pretty reasonable idea.

So, I started working on that. As of last night, I’ve sketched out two small libraries, one for RSS feeds and one for the competing standard, Atom. These libraries are thin wrappers around the XML library to deal with the specifics of RSS and Atom. Here’s an example of consuming an RSS feed:

var rss = Rosella.Rss.read_url("http://www.parrot.org/rss.xml");
rss
    .channels()
    .first()
    .items()
    .foreach(function(i) {
        say(Rosella.String.format_obj("{title} (by {creator}) : {description}", i));
    });

You can do almost exactly the same thing with an Atom feed too, if you’ve got one of those instead. Right now RSS and Atom are implemented in two separate libraries, but I may combine them together for simplicity and to avoid unnecessary code duplication.

I’m working on an interface to write and publish feeds as well, though that’s not quite ready yet. You can bet that when I’ve got that working, I’ll be setting up a copy of mod_parrot to use it with.

I’ve been sort of kicking around the idea of a specialized HTML parsing library, which would more or less be an SGML parser with some schema information. I’m not sure I want to get into that hassle because HTML is a pretty messy thing and it will take a huge amount of effort to get something that works most of the time. But, if you’re willing to put up with a little bit of oddity, the Xml library works well enough for many cases.

parrot.org: PACT: Time flies when you're writing code.

benabik — 2012-06-28T22:58:24+00:00

How many days are in a week? Judging from my weekly blog posts, there are 14 days in a week. *sigh* Well, I knew my schedule was going to be a little erratic this summer but apparently underestimated slightly.

Now, to be fair, I'm actually not all that far behind schedule. It might have looked that way over the last couple of weeks, but that's because I tend to hold onto code and continue to revise commits until I have a large chunk of functionality working.

Andrew Whitworth: Sockets and Encodings

2012-06-28T07:00:00+00:00

The io_cleanup1 branch is nearing completion, though as always the last few details are what holds everything up. In the past few days all the remaining tests in the parrot repo were passing. The coding standards tests, as usual, the last to be resolved. Then I started building and testing other things on the branch: Winxed builds and tests fine. So does Rosella. Then I looked at NQP and Rakudo. Both built fine, but Rakudo was failing two socket-related spectests.

That’s not entirely unexpected. Even though my intention was to make this branch as painless as possible there were still some unavoidable changes to interfaces and semantics. There are a few places where older semantics are surrounded by large /* HACK! */ comments, but for the most part I’ve tried to make everything sane. That’s why I wasn’t surprised to see Rakudo failing a few tests. I was much more surprised that Rakudo built without any problems the first time I tried it. I figured the test failures represented some kind of semantic mismatch, and getting Rakudo passing again would have been as easy as getting the old semantics returned, with a note about a future update path.

It turns out this wasn’t exactly the case. For one test it was the simple difference in the way we read on streams with multibyte encodings. This was expected and we can fix it to use the old behavior if that’s what Rakudo prefers. For the second failing test, it’s not that there’s a semantic difference per se, but instead there is a glaring and serious bug in master that was corrected in the new branch. Here, I’m going to explain what’s going on.

Look at this code:

Parrot_io_recv_handle(PARROT_INTERP, ARGMOD(PMC *pmc), size_t len)
{
    Parrot_Socket_attributes * const io = PARROT_SOCKET(pmc);

    /* This must stay ASCII to make Rakudo and UTF-8 work for now */
    STRING * res    = Parrot_str_new_noinit(interp, len);
    INTVAL received = Parrot_io_recv(interp, io->os_handle,
                                     res->strstart, len);

    res->bufused = received;
    res->strlen  = received;

    return res;
}

This is a pared-down version of the code behind the recv method on Socket. It creates a new string with the specified length pre-allocated, then passes the buffer to the low-level recv C API (which has been abstracted a little to account for platform differences).

Notice the comment there in the middle which says the string uses the ASCII encoding, for use by Rakudo. This is what I saw, and this is the semantic I followed in the new system: When you read from a socket by default in the new system, the string is encoded as ASCII unless you specify differently.

Just for my own verification, I had to look at the Parrot_str_new_noinit function to verify that the string was, in fact, being set to ASCII:

Parrot_str_new_noinit(PARROT_INTERP, UINTVAL capacity)
{
    STRING * const s = Parrot_gc_new_string_header(interp, 0);
    s->encoding = Parrot_default_encoding_ptr;

    Parrot_gc_allocate_string_storage(interp, s,
        (size_t)string_max_bytes(interp, s, capacity));

    return s;
}

Elsewhere in the system, we have this:

Parrot_default_encoding_ptr = Parrot_ascii_encoding_ptr;

So yes, the string returned by the Socket does indeed use the ASCII encoding in master. And, after double-checking, the version in the io_cleanup1 branch was using ASCII also. However, in the new branch Rakudo’s test fails because of an exception about a lossy conversion of non-ascii data into the the lower bit-width format. A quick check shows that both systems create an ASCII string buffer and both systems call the same recv function to fill it. So where’s the problem? What the hell?

For comparison, here’s the snippet of code from the new branch that reads data into a STRING, possibly using a buffer:

bytes_read = Parrot_io_buffer_read_b(interp, buffer, handle, vtable,
                                   s->strstart + s->bufused, byte_length);
s->bufused += bytes_read;
STRING_scan(interp, s);

We’re reading out a number of bytes, appending them into the string’s pre-allocated storage and updating the number of bytes actually used. That’s all the same as in master. However, the last line, STRING_scan does not appear in master. What is it?

STRING_scan() loops through the data in the string to verify that it correctly matches the string’s encoding. For instance, if the string is encoded as ASCII, STRING_scan will loop through to make sure all character values are lower than 128. If the string is UTF-16, STRING_scan verifies that we have an even number of bytes and that each value is an acceptable codepoint.

master doesn’t do this, which means there is a bug. In master, we don’t scan the string after recv but before we return it to the user, which means we can have non-ASCII data in a string marked with the ASCII encoding. The Rakudo test puts UTF-8 data into the socket on the server side, and then reads out a string and encodes that to UTF-8 to verify that it comes out correctly. However in the new branch we actually check that the string is valid before giving it out to user code, and it isn’t, so we throw an exception.

Combine that with the fact that the Socket PMC has no way to change the encoding it uses in master, which means all Sockets used in Parrot master are potential sources of bugs.

Two nights ago I added methods to Socket to get/set the encoding to use, and everybody’s favorite Moritz created a branch for Rakudo to use it. Last night I did some playing with default encodings. Tonight and into the weekend I’m hoping to wrap up the last few details to get the Rakudo spectest passing like normal again. Hopefully, if all goes well, we can start talking about a merger within the next week or two.

parrot.org: Progress report

jharper1 — 2012-06-26T15:35:20+00:00

I have been working on the api.c file that handles the functions for the security. Revising and editing functions with the help of Whiteknight and Dukeleto. Right now I am allocating, initializing and freeing memory for the functions as well as integrating the api.c and utility.c files I am working on into root.in. I expect to get much done this week in terms of the api.

parrot.org: What have we done for you lately

brrt — 2012-06-25T09:04:13+00:00

And by we, I mean myself, parrot, and mod_parrot. That is simple: cgi-style running LIVES AGAIN (almost, just need to fix headers ;-)). And with it, all the infrasturcture to implement more and nicer loaders, such as those for PSGI and / or WSGI, and the famous inline loader-in-the-sky I will be writing. Pretty nice, no?

For the technically interested, what has happened is that:

Loaders now accept 3 arguments: the request (as a PtrBuf). This is an opaque handle, that can be used to bind to the apache input / output handles.

parrot.org: 4.5.0 Parrot "Buff-faced Pygmy Parrot" Released!

ayardley — 2012-06-19T21:29:45+00:00

In honor of M0: "The tiny bird, which is not much bigger than an adult
person's thumb, is smaller than some of the insects with which it shares the
forest. On average, buff-faced pygmy parrots (Micropsitta pusio) stand less than 9cm tall and weigh 11.5g (0.41oz)" --BBC Earth News.

read more

parrot.org: Testing Part

Jashwanth — 2012-06-18T16:25:03+00:00

This week was spent on fixing the errors encountered basically by adding get_pointers() functions to some of the files in parrot and pla.Then later on started to write tests on the project to check the correctness of the result from lapack subroutines but have to change the implementation of the initial file so that it would be helpful for the testing part.Had also encountered and learned about an error from git regarding merge conflict.

parrot.org: Small Progress in Security Sandbox

jharper1 — 2012-06-14T13:16:35+00:00

Started the project off with a small problem. Parrot refused to build correctly on my windows machine. In the end I replaced it with Ubuntu to prevent falling behind further in my projected time line and have made little progress. It seems the biggest problem with developing the core for me is Parrots internals and fully understanding them. Whiteknight has been patient and giving guidance. I plan on submitting a small list of what exactly I am having problems with hopefully sometime today so I can prevent further snags. I welcome any and all advice.

parrot.org: Update on infrastructure

brrt — 2012-06-13T18:49:48+00:00

All things considered mod_parrot is a little less than a 1000 lines long. About half of that is infrastructure (building, testing). For that, it now buys you the ability to configure, start and stop a mock apache server via the pudding framework (testing is influx right now... Karma for whoever can guess why its called pudding.). Not only that, I can attach gdb to the same, also via pudding.

Andrew Whitworth: Reading a Line of Text

2012-06-13T07:00:00+00:00

In terms of usage, there aren’t too many IO-related features in Parrot’s user interface more straight-forward than the readline method. It does exactly what you tell it to do: read a line of text from the given file and return that line of text as a Parrot string. Easy.

Tonight I was looking at some of the old code to get an idea about expected semantics for some tests that need fixing. Let’s look at some code:

.sub read_a_line
    .param string type
    $P0 = new [type]
    $S0 = $P0.'readline'()
    .return($S0)
.end

.sub test_readline
    $S0 = 'read_a_line'('FileHandle')
    say $S0
    $S0 = 'read_a_line'('Socket')
    say $S0
    $S0 = 'read_a_line'('StringHandle')
    say $S0
.end

The valid types for this are, as usual, "FileHandle", "Socket" and "StringHandle". Notice that we’re reading a line from the object of the given type before we’ve opened, connected or initialized. Pretend, in order to save myself some typing, that I’ve set up exception handlers and the like above. So, what happens?

For FileHandle we throw an exception. You can’t read from a closed handle.
For StringHandle, we throw an exception for the same reason.
For Socket we return null because…whatever. (in the test suite we test that when converted to a floating-point number, that it’s 0.0. Again, whatever).

So that’s a little bit weird that socket does something different from the other two, but fundamentally it’s a pretty different type so I suppose some differences can be allowed.

Now, let’s try something slightly different:

.sub read_a_line
    .param string type
    $P0 = new [type]
    $P0.'open'("foo.txt", "r")
    $P0.'print'("This is \n test text")
    $P0.'close'()
    $S0 = $P0.'readline'()
    .return($S0)
.end

With this example we can only operate on FileHandle and StringHandle because Socket doesn’t have an .open() method like those two do. What does this do for those two types?

For FileHandle we throw the same exception, you still can’t read from a closed handle.
For StringHandle you can read like normal without any indication that the handle is closed!

So that’s weird to say the least that StringHandle has two different behaviors. Socket has yet another problem, in a slightly different way. The method Socket.readline() returns null when not open, but if you pass a Socket to the Parrot_io_readline method, it always throws an exception because apparently readline on a Socket isn’t supported! And because readline on a Socket uses a completely different code path from FileHandle the two types use completely different buffering mechanisms with subtly different semantics (StringHandle, because it uses the in-memory string buffer, does it in a third way).

To recap: What is conceptually a simple operation, read in some text until we find a delimiter, is done in three completely different ways by three different types, each with different error-handling semantics depending on both history, state, and the interface used. If anybody was wondering why I wanted to rewrite this subsystem, here’s part of the reason.

Actually, I kind of lied. It’s really not a simple operation which is all the more reason we should share common code. It’s a clear case of an algorithm where the hard parts should be encapsulated inside a clean interface so that different types can avoid needing to reimplement it over and over again (with differences, bugs and complications). That’s the way it really should be, but some of the complications in the code are a little hard to live with. Here’s the general algorithm for readline on a FileHandle, as it’s implemented in Parrot master:

The filehandle requires a buffer for this, so create (and fill) a buffer if one isn’t configured.
Create a new, empty STRING header.
Treating the buffer like an encoded STRING, scan the buffer looking for the end of the delimiter or the end of the buffer, whichever comes first.
Allocate/reallocate enough space in the STRING header to hold all the data we’ve found in the buffer.
Append all the characters we’ve found to the STRING.
If we’ve found the delimiter, we’re done. Return it to the user.
Otherwise, check if we are at the end of file for the input. If so, go to 8. If not end of file, go to 9.
Check that the last codepoint is complete and has all its bytes. If so, return the STRING to the user. If not, throw an exception about a malformed string.
Check that the last codepoint is complete and has all its bytes. If so, go to 10. Otherwise, go to 11.
Refill the buffer and go to 3.
Determine how many more bytes we need to read to complete the last codepoint.
Refill the buffer, and check that we have at least that many bytes available to read. If so, go to 13. Otherwise, throw an exception about a malformed string input.
Read in the necessary number of bytes (1, 2 or 3 at most) from the buffer and go to 3.

If you’re reading an ASCII or fixed8 string the logic obviously collapses down to something a little bit more manageable. Also, this same logic, almost line for line, is repeated in the routine to read a given number of characters from the handle, where characters in a non-fixed-width encoding (like utf8) may need multiple reads to get if we don’t get all the bytes for the character into the buffer in a single go. Notice that the versions provided by StringHandle and Socket are both much more simple and not safe for multi-byte encodings like utf8 or utf16.

In my io_cleanup1 branch, the logic has been simplified substantially, and a single codepath is now used for all three of the major types:

Make sure the handle has a read buffer set up and filled.
Create a new, empty STRING header.
Ask the buffer to find the given end-of-line character. The buffer will return a number of bytes to read in order to get a whole number of codepoints, and a flag that says whether we’ve found the delimiter or not.
Append those bytes to the string header.
If the delimiter is found or if we are at EOF, return the string.
Fill the bufffer and go to #3.

By simply coding the buffer logic to refuse to return incomplete codepoints in response to a STRING read request, the whole algorithm becomes hugely simplified. The readline routine in master takes up 185 lines of C code. In my new branch, the same routine takes up only 47 lines. Of course, this isn’t comparing apples to apples, because I did break up some of the repeated logic into helper routines, and the buffers in my system are obviously a little bit smarter about STRINGs and codepoints, but that’s not exactly the point. The real point is that three large, complicated, hard-to-read functions in master are now a single, much smaller, easier-to-read routine that relies on clear abstraction boundaries to do a difficult job in a much more conceptually simple way.

I’ve also updated the STRING read routine (now called Parrot_io_read_s) to use a similar algorithm and actually share some of the new helper methods. That sharing itself also helps to decrease total lines of code has has other benefits as well.

Notice that there is one small change in these two algorithms, which may or may not need to be worked around if it causes problems. Notice that we don’t read out of the buffer an incomplete codepoint. If we have an incomplete one at the end of the file, the first algorithm will read it in and throw an exception about a malformed string. The second algorithm will ignore those final bytes and successfully return all the rest of the valid-looking data from the buffer instead. In the first algorithm, it then becomes impossible to read the partial data out and make a best effort, while in the second algorithm you can easily get to the data, even if the last codepoint is corrupted and cannot be read. I’d really love to hear what people think about this change, and whether it’s worth keeping or needs to change. I suspect it is better this way but only the users can really say for sure.

parrot.org: PACT: Infrastructure

benabik — 2012-06-12T19:02:09+00:00

Oh, right. Weekly blog posts. benabik-- In the same style as last year, I'm going to include what I said I'd do and what I have done.

May 30 Improvements to Key PMC: Creation/introspection of keys with register contents. This both a useful improvement to Parrot on its own and useful for later portions of this project.

parrot.org: The initial step

Jashwanth — 2012-06-11T17:35:31+00:00

This is about the project "LAPACK binding with PLA".
To implement the project have been using winxed,read some code and examples related to this on http://whiteknight.github.com/Rosella/winxed/index.html
and have started to code in the same.
Till date have am trying to call the LAPACK function by using some of the matrices in PLA but got an error regarding a pointer and trying to figure out the cause as everything being passed to the lapack function is a pointer.

Andrew Whitworth: IO Rewrite Status

2012-06-08T07:00:00+00:00

I was going to call this post “IO Cleanup Status”, but let’s face facts: This is a complete rewrite of the entire subsystem. I haven’t hardly left a single line of code untouched. It is a full rewrite of the system hiding behind a mostly-similar (though not quite the same) API. I didn’t intend to completely rewrite the whole subsystem when I started the branch, hence the benign-sounding branch name. Following along with our cultural norms, I could have called it whiteknight/io_massacre or something similarly upbeat. Whatever. I’ve known people stuck with un-liked names for their entire lives, so this branch can be misnamed for a few weeks.

So what is the status of this branch, exactly?

At the time of this writing the branch is mostly complete. The major architectural work has all been done, with per-type logic separated out into new IO_VTABLE structures, and buffering logic divorced from FileHandle into a new IO_BUFFER structure. Now you can do things that have never been possible before, like buffering socket input and output, or doing readline with custom line-end characters on all handle types, and a whole bunch of other, increasingly-obscure operations. A lot of the new capabilities are things you didn’t even know we didn’t support before. Now, we do.

We aren’t quite there yet, but the stage is set for some other awesome changes in the future too, which I’ll talk about in more depth when we get there.

The current status of the branch is good. Parrot builds without any huge amount of new warnings and with no errors on my platform. Some platform-specific code needs to be updated for Windows, I’m sure. The one big thing standing in the way is keeping track of file positions through operations like seek and tell. These things are made a little bit more difficult when you have read buffers reading ahead, because the position of the next character to read according to the user may be far different than the position of the file descriptor according to the operating system. Then consider the case when you have a file opened for read and write, with buffers in both directions. The old system had a single buffer per FileHandle which needed to be flushed if you tried to read when the buffer was in write mode, or you tried to write when it was in read mode. If you’re switching back and forth between reading and writing often enough, buffering actually decreases performance when it’s supposed to be a performance enhancer.

The FileHandle has an attribute to keep a pointer to the current cursor location, but I’m not always updating it as often as I should and not always reading it when I should. If you have a file opened for read and write, when you write 5 characters at the current file position you need to increment the read buffer by 5 characters also. When you go to read in 5 characters from the current position, you either need to flush the write buffer first or you can try to read those characters right out of the write buffer. There’s nothing complicated about it, just a lot of bookkeeping to get right and lots of little interactions that need to be tested. It’s helpful that we don’t do seek or tell on some things like Sockets, and we don’t really buffer StringHandles.

The branch is moving along well and if I can find the time to actually sit down and work on it for a dedicate period of time I might be able to get it closer to being done. I’m shooting for being mergable sometime after the coming release.

parrot.org: A few steps forward

brrt — 2012-06-03T16:33:40+00:00

So, what happened in the world of mod_parrot? Segfaults, thats what happened. And an awesome community helping and fixing every problem.

First the segfault. What happened is that I called compreg() from within a script. That failed because the compreg opcode assumes a hash to initialized, which wasn't. (It was NULL). So once I mentioned this on the mailinglist, within minutes whiteknight++ pushed a fix that made the opcode use the proper API function which fixed the problem.

Andrew Whitworth: IO Cleanups Status

2012-06-01T07:00:00+00:00

Tonight I’ve hit something of a milestone with my branch to rewrite the IO subsystem. As of tonight the parrot binary, parrot-nqp and winxed all build in my branch and coretest runs (though fails some tests). The entire build does not complete because of some failures related to dynops, but it does get most of the way through. This means that most of the main-path IO APIs and FileHandle operations are working correctly, which is a relatively small portion of everything that has changed.

With Parrot building, I’m now able to more closely keep track of progress and regressions, and do more live testing as I make new changes. Until this point all my changes have just been mental exercises, so I’m happy to have a little bit more feedback and even some validation.

Of course, just saying that it builds doesn’t really mean anything. Several things are still not implemented or completely wired up. Some operations on files such as seek, peek and tell are still not implemented yet. Several methods on the various PMCs (FileHandle, Socket and StringHandle) have not been updated to use the new system. There are a few regressions I need to address with regards to buffering. Specifically, “line buffering” has been removed from the system during the rewrite and hasn’t been added back. Line buffering in Parrot has never really done much, but it’s just hacky and obscure enough that I’m sure somebody is relying on it.

Some things, like files opened for dual read/write modes or append modes haven’t been completely dealt with in code either. I don’t think there’s a lot of work to do for this, but since the buffering architecture has changed so much from what it used to be and since these modes are relatively rare and not as thoroughly tested I want to spend a little bit of extra time making sure there are no regressions.

Also there are several coding standards tests (especially for function-level annotations and documentation) which fail spectacularly in the branch, and it’s going to take time to update all the old documentation and add docs for all the new functions. I also need to update PDD 22 to reflect the new architecture of the IO system.

I’ve been working on this branch pretty aggressively for the last two weeks and I think I’m about 50% of the way done. That’s not too bad considering the magnitude of the change and the amount of time I’ve had to hack. Within a week or two more, if all goes well, I think the branch might be ready for wider testing and eventually merging.

As usual when we’re talking about changes this big, merges are not something to be rushed. Assuming all goes well and other people like what I’ve been doing, expect to see a brand new IO system in Parrot sometime later this summer.

Andrew Whitworth: IO Refactors

2012-05-27T07:00:00+00:00

The IO subsystem is a lot like the garbage collector: So long as it just works we can ignore its faults for quite a long time. The garbage collector had performance and other issues for years before everybody’s favorite bacek went through and finally rewrote it. His effort there saves the rest of us meer mortals from having to touch the GC again for another couple years.

The IO system works reasonably well. It’s got a decent set of features more or less, it implements most of the important operations that our users have needed in the past, it’s not spectacularly slow (and disk or network operation performance almost always outweighs any issues in the code that leads to those things), and we haven’t been getting a lot of error reports or feature requests for it. In short, if it ain’t broke, don’t fix it.

A few days ago I was working on a ticket for moritz to add better integration between our various IO vector PMCs (FileHandle, Socket, etc) and the ByteBuffer PMC. ByteBuffer is what it’s name implies: It’s an array-like type for working with individual bytes in a chunk of memory. It’s like a binary encoded STRING, but it’s not immutable and has a handful of additional features that a raw STRING (or the String PMC) doesn’t. ByteBuffer can be populated from and exported to a STRING, and it is useful for certain types of operations that need to operate on a sequence of bytes without having to worry about strings and encodings and all that other nonsense. Mortiz’s request was a reasonable one so I sat down and made it happen. A few nights ago I merged that work in to master with an “experimental” tag on it.

However while I was in the IO subsystem code making this happen something did break. Not in the code, instead something broke inside my poor little head. The snapping sound you hear is the poor camel’s back under the load of that last piece of straw. I’ve had enough of that system and its inside-out organization and collection of half-ideas and botched refactors. I’ve had my fill of the nonsense and finally decided it was time to make things right.

And before anybody says to me, “hey Mr Whiteknight, you shouldn’t be so mean, somebody probably worked really hard to make this code do what it does”, let me just say two things: First, “Mr Whiteknight” is my father’s handle and Second, I was one of the people who helped put IO where it is today. I don’t feel particularly bad insulting myself or my own work, and my contributions, though well-intentioned at the time, are a big part of why the system is in the condition its in now. First, a brief history lesson.

When I joined Parrot, it sported an IO system based on layers. Layers were arranged in a structure something like a vtable, and IO requests would be fed through the layers. Each layer getting the output of the one before it until the bottom layer actually spat the data out (or, read it in depending on which way you were moving). This worked pretty well when you were trying to do File IO on a file with a particular encoding, with buffering, through an asychrony mechanism, etc. Actually I say it worked well but it was sort of overkill: It was just too much infrastructure for the possible benefits and despite the theory of allowing better code reuse there really weren’t too many different layering combinations that could be set up. Plus, layers start to interdepend and violate encapsulation, then optimization starts prompting a few “short cuts” where layers were flattened together. One of the earlier things I did on Parrot, post-GSOC, was to remove some of the last vestiges of the then-unused layering system from Parrot’s IO.

The IO subsystem has something of a problem where it has a few masters and has to be performance conscious. Many of our programs are still the kind that shuffle data about (very much in the influence of Perl) and IO operation performance mattered when your compiler is reading in HLL code and outputting PIR code, then you’re reading PIR code in and trying to compile it again. Too much nonsense and everybody feels it.

In Parrot at the user level you can do IO in two ways: Through the IO PMCs (FileHandle, mostly) and through opcodes (say, print, etc). The problem, put succinctly, is this: We want to encapsulate logic for writing to files inside the FileHandle PMC, but we don’t want to add new IO-specific VTABLES and we don’t want to incur the costs of method calls on every single IO request. In other words, we didn’t want the print opcode to just be a thin wrapper around the print method on FileHandle. Such a thing, especially if implemented naively, would have killed performance by creating nested runloops and a whole host of other problems.

The way the system is set up is that both FileHandle.print() and the print opcode are both thin wrappers around the real routine Parrot_io_putps, which does all the hard work. And, more importantly, that routine is expected to act transparently (like the print opcode does) on any IO PMC type like Socket or StringHandle. The only real way to do this, if you can’t call a method on the FileHandle and Socket PMC is to use a large switch-statement:

switch (handle->vtable->base_type) {
    case enum_class_FileHandle:
        ...
    case enum_class_Socket:
        ...
    case enum_class_StringHandle:
        ...
    default:
        Parrot_pcc_invoke_method_from_c_args(..., handle, "print", ...);
}

I’ve obviously glossed over all the details, but this is the general form of that routine and several other similar routines in the IO API. You’ll notice several things from even a quick glance at this example:

If we want to add a new IO type to Parrot core we need to add a new entry to the switch statement in every IO API routine that needs to care about PMC type (this is a major part of the reason we don’t yet have a sane, separate Pipe type).
If the user passes in an Object, something defined at the PIR level, we do fall back to calling the method, because we can’t do anything else intelligently.
We can’t really subclass FileHandle or Socket from the user level, because it would fail the base_type test, and wouldn’t be able to handle the low-level structure accesses from that point forward anyway.

Point number 2 is particularly interesting because the FileHandle.print() method calls Parrot_io_putps, which may turn around and call the .print() method. This is a big part of the reason why FileHandle cannot be subclassed in user code. It’s clearly an example of poorly separated concerns and poor encapsulation. Either the method should call the IO API or the IO API should call the method but we can’t be doing both. Actually, I’d far prefer the former, if we can do it in a good, general way.

There are a few other issues worth mentioning, which I’ll just dump rapid-fire without much explanation:

We don’t have a separate Pipe type. Instead, FileHandle can be opened in “pipe mode” to write to a separate process or read output from a separate process.
We have limited buffering, but only on FileHandle and we cannot configure buffers for input and output separately, or use separate buffers.
We don’t really have encodings set up in any consistent way, so it’s very possible, though I haven’t worked out all the details, to write strings with different encodings to a file. This is especially true if we’re using buffers and performing writes through different API routines.
FileHandle logic is considered to be the default and is given deference in the code. Pipe logic is unified with file logic at a very low level. Socket and StringHandle are treated as bolted-on spare parts and don’t benefit from hardly any code sharing or unified architecture. They also don’t have all the same useful features as FileHandle has.
Several functions in the IO subsystem are poorly or inconsistently named and implemented, not to mention the often-times confusing documentation and absurd architectural arrangements.

So that’s the system we’ve got. What do I want to do to fix these issues?

The first thing I’ve suggested is to break up IO functionality into an IO_VTABLE of function pointers, similar to how the STR_VTABLE, the sprintf dispatch mechanism, the packfile segment dispatch table and other similar mechanisms in Parrot work. Each IO request would go through the API routines, which dispatch to a vtable routine (possibly with some intermediate buffering logic). Here’s what it looks like in the branch to do a basic write:

IO_VTABLE * const vtable = IO_GET_VTABLE(interp, handle);
vtable->write_s(interp, handle, str->strstart, str->bufused);

And here’s how to do it with write buffering:

IO_VTABLE * const vtable = IO_GET_VTABLE(interp, handle);
IO_BUFFER * const read_buffer = IO_GET_READ_BUFFER(interp, handle);
Parrot_io_buffer_write_s(interp, handle, vtable, buffer, str);

Internall, the buffer does it’s magic and flushes data out to the vtable if necessary.

The second thing I want to do is break out buffering so that instead of being a detail of the FileHandle PMC a buffer is a separate struct which can be attached to any IO type as desired. And, even better, we can attach multiple buffers to an IO stream, at least one each for input and output, configured separately. The buffering API, which will be cleaned up and properly encapsulated, will take a pointer to the IO_VTABLE for the handle and will pass data through transparently as required. A thin wrapper PMC type, IO_BUFFER, would allow references to buffers to be accessed and configured directly, which would be very useful in some cases.

Imagine, if I may go off on a short tangent, a threaded system where one worker task had a reference to a buffer and continuously made sure it was filled in the background while another worker task read bits and pieces from the buffer very quickly. It would be possible, through careful choice of algorithm, to do such a thing lock-free. Feel free to replace “file” with “socket” or “pipe” in the example above too. Imagine also a system where we can transparently use mmap (or it’s windows equivalent) to map a file to memory as part of the buffer, and keep working with it that way.

The third thing I want to do is start teasing apart the logic for Pipes from the file logic. I’ll create a separate io_vtable for pipe operations, and use that inside FileHandle when we’re in pipe mode. Eventually we’ll be able to create a separate type, divide out all the logic completely, and get to work on really interesting stuff like feasible 2-way and 3-way pipes.

The fourth thing I want to do is start setting up interfaces so that IO operations including buffering, low-level IO, file descriptor manipulation and other things become more accessible at the PIR level so users can make better use of these tools, both in subclasses of the in-built handle PMCs and in custom types which neither derive from nor hold instances of those types.

I’ve started sketching out many of these ideas in the whiteknight/io_cleanup1 branch. cotto seems to agree with the general direction and I haven’t heard any complaints so far, so I’ve had my head down and been working hard on making these ideas reality. As of this writing, I’ve modified just about every single line of code in the subsystem, gotten most of the new architecture and logic into place and set up the vtables for the most important built-in types. I have a few details to finish up before I try to build (and inevitably debug) this new beasts. Ultimately I would like this first round of cleanups to produce no user-visible changes, so the old PMC methods and exported API functions are going to continue doing what they’ve always done. Later rounds of cleanups will add new interfaces and eventually deprecate and remove some of the crufty older ones. I’ll post more updates as this work progresses.

Andrew Whitworth: Destructors are Hard

2012-05-23T07:00:00+00:00

««<« HEAD:drafts/gc_destructors.md In my last post I mentioned some of the work I was trying to do with GC finalization and destructors. I promised I would publish a longer and more in-depth post about destructors, what the current state is, what I am doing,

and what still needs to be done.

In my last post I mentioned some work involving the GC, finalization and destructors. Today I’m going to expand on some of those ideas, talk about what the current state of destruction and finalization are in Parrot, some of the problems we have with coming up with better solutions, and some of the things I and others are working on to get this all working as our users expect us to. I apologize in advance for such a long post, there’s a lot of information to share, and hopefully a much larger architectural discussion to be started.

3d0a502723cb7124eb717d7e82bac5ecc567ac31:_posts/2012-05-23-destructors_are_hard.md

Destructors are hard. The idea behind a destructor is a simple one: We want to have a piece of code that is guaranteed to execute when the associated object is freed. Memory allocated on the heap is going to get reclaimed en masse by the operating system when the process exits. However, things such as handles, connections, tokens, mutexes, and other remote resources might not necessarily get freed or handled correctly if the process just exits, or if the object is destroyed without some sort of finalization logic performed on it. Here’s a sort of example that’s been bandied about a lot recently:

function main () {
    var f = new 'FileHandle';
    f.open("foo.txt", "w");
    f.print("hello world");
    exit(1);
}

In this example we would expect that the text "hello world" would be written to the foo.txt file. However, because the text to be written may be buffered (both in Parrot and by the OS), there’s a very real chance that the data won’t get written if we do not call the finalizer for the FileHandle PMC.

Obviously, the brain-dead solution to this particular problem is to manually close or flush the file handle:

function main () {
    var f = new 'FileHandle';
    f.open("foo.txt", "w");
    f.print("hello world");
    f.close();
    exit(1);
}

However, the whole point of having things like finalizers (“destructors”) and GC is to make it so that the programmer does not need to worry about little details like these. The program should be smart enough to find dead objects in a timely manner and free their resources. Beyond that, many programming languages (with special emphasis on Perl6) require the availability of reliable and sane destructors.

In the remainder of this post I would like to talk about why destructors are hard to implement correctly, why Parrot does not currently (really) have them, and some of the ideas we’ve been kicking around about how to add them.

First, let’s cover where we currently stand. Parrot does have destructors, of a sort, in the form of the destroy vtable. That routine is called by the GC when the object is being reclaimed, during the sweep pass. A side-effect of this implementation is that if PMC A refers to PMC B and both are being collected, it’s very possible that A’s destructor tries to access some information in B after B has already been reclaimed. Think about a database connection object that maintains a socket on one side, and a hash of connection information on the other. The socket probably cannot perform a simple disconnect, but instead should send some sort of sign-off message first to alert the server that it can proceed with its own cleanup. The socket PMC would need information from the connection information hash to send this final message, but if the hash had already been reclaimed the access would fail with undefined results.

This situation has lead to more than a few calls for ordered destruction. In one of the most common and severe cases, Parrot’s Scheduler PMC was being relied upon by various managed PMCs. When a Task PMC was destroyed, at least in earlier iterations of the system, it would attempt to send a message to the Scheduler that it was no longer available to be scheduled. Ignore for a moment the fact that the Task could not possibly have been reclaimed in the first place if the Scheduler had a live reference to it, and if the Scheduler was still alive itself.

Because of some of these order-of-destruction bugs, GC finalization (a final, all-encompassing GC sweep path guaranteed to execute all remaining destructors prior to program exit) had been turned off. That and performance reasons. Turning off GC finalization leads to the problem above where data written to the FileHandle is not not flushed before program exit. You are probably now starting to understand the bigger picture here.

Having ordered destruction means essentially that we should be able to have an acyclic dependency graph of all objects in the system with destructors. However, maintaining this in the general case is impossible and attempting to approximate it would be very expensive in terms of performance. In any case, this is just a way to work around the problem of our naive sweep algorithm, which destroys and frees dead objects in a single pass, and not a real solution to the larger problems. A far better idea, recently suggested by hacker Moritz, is a 2-pass GC sweep.

In the 2-pass case the GC sweep phase would have two loops: the first to identify all PMCs to be swept (from a linear scan of the entire memory pool), execute destructors on them and add them all to a list, and the second to iterate over that list (after all destructors had been called) and reclaim the memory. Because of the linked-list setup of the GC, this second pass could, conveivably, be almost free because we could simply append this list of swept items to the end of the free list for an O(1) operation , and the first pass would be no less friendly on the processor data cache than our current sweep would be. This, in theory, solves our problem with ordered destruction, and should allow us to re-enable GC finalization globally without having to worry about these kinds of bugs causing segfaults in the final milliseconds of a program.

So that’s the basics of our current system and our problem with GC finalization, and shows us how we would proceed to make sure destructors were always called as a guarantee of the VM. However, this doesn’t begin to address any of the problems with destructors that will plague their implementation and improvement. I’ll talk about that second subject now.

Destructors, as I said earlier, are hard. In the case of GC finalization, after the user program has executed and exited, it’s relatively easy to loop over all objects and call destructors. It is those destructors which happen during normal program execution that cause problems.

In the C++ language, destructors have certain caveats and limitations. For instance, we can’t really throw exceptions from destructors, because that may crash the program. Not just an “oops, here’s an exception for you to handle”, but instead a full-on crash. In Parrot we can probably be smarter about avoiding a crash but not by much. It’s a limitation of the entire destructors paradigm. Let me demonstrate what I’m talking about.

Let’s say I have this program, which opens up a filehandle to write a message and then starts doing something unrelated to the filehandle but expensive with GC:

function foo() {
    var f = new 'FileHandle';
    f.open("foo.txt", "w");
    f.write("hello world!");
    f = null;       // No more references to f!

    for (int j = 0; j < 1000000; j++) {
        var x = new MyObject(j);
        x.DoSomething();
        x.DoSomethingElse();
        x.DoOneLastThing();
    }
}

Somewhere along the line, when the GC runs out of space, it’s going to attempt a sweep and that means that f is going to be identified as unreferenced, finalized and reclaimed. The question is, where? The thing is that we don’t know where GC is going to run for a few reasons:

We don’t know how many free headers GC has left in the free list before it has to sweep to find more.
We don’t know how many PMCs are being allocated per loop iteration, because the various methods on x could be allocating them internally, and all PCC calls currently generate at least one PMC, and this is a lot of pressure.

So at any point in that loop, any point at all, GC could execute and reclaim the FileHandle f. That calls the destructor, flushes the output, and frees the handle back to the operating system. Good, right? What if there is a problem closing that handle, and the destructor for FileHandle tries to throw an exception (or, if this example isn’t stoking your imagination, imagine that f is an object of type MyFileHandleSubclass with an error-prone finalizer).

There are a few options for what to do here. The first option is that we throw the exception like normal. This means that the loop code with the MyObject variables, which is running perfectly fine and has no reason to throw an exception by itself, is interrupted in mid loop. The backtrace, if we provide one at all, probably points to MyObject but with an exception type and exception message indicative of a failed FileHandle closing. Initial review by the poor developer doing the debugging will show that there are no filehandles trying to close inside this loop and then we get a bug report because a snippet of code which is running just fine exits abruptly with an error condition which it did not cause. The solution for this, wrapping every single line of code you ever write in exception handlers to catch the various possible exceptions thrown from GC finalizers, is untenable from a developer perspective.

A second option is that we somehow disallow things like exceptions from being thrown from destructors, because there’s no real way to catch them rationally. This seems reasonable, until we start digging into details. How do we disallow these, by technical or cultural means? And if we’re relying on cultural means (a line in a .html document somewhere that says “don’t do that, and we won’t be responsible if you do!”), what happens if a hapless young programmer does it anyway without having first read all million pages of our hypothetical documentation? Does Parrot just crash? Does it enter into some kind of crazy undefined state? Obviously we would need some kind of technical mechanism to prevent bad things from happening in a destructor, though the list of potentially bad things is quite large indeed (throwing exceptions, allocating new PMCs, installing references to dead about-to-be-swept objects into living global PMCs, etc) and filtering these out by technical means would be both difficult and taxing on performance. When you consider that even basic error reporting operations at an HLL level, depending on syntax and object model used, may cause a string to be boxed into a PMC, or a method to be called requiring allocation of a PMC for the PCC logic, or whatever, we end up with finalizers which are effectively useless.

A third option is that we could just ignore certain requests in finalizers, such as throwing exceptions. If an exception is thrown at any point we just pack up shop, exit the finalizer and pretend it never happened. This works fine for exceptions, but does nothing for the problem of a finalizer attempting to store a reference to the dieing object into a living object. I don’t know why a programmer would ever want to do that, but if it’s possible you can be damned sure it will happen eventually. Also, when I say “pack up shop”, we’re probably talking about a setjmp/longjump call sequence, which isn’t free to do.

The general consensus among developers is that errors caused by programs running on top of Parrot should never segfault. If you’re running bytecode in a managed environment, the worst that you should ever be able to get is an exception. Segmentation faults should be impossible to get from a pure-pbc example program.

However, as soon as you introduce destructors, suddenly these things become possible. And not just from specifically malicious code, even moderately naive code will be able to segfault by storing a reference to a dieing PMC in a place accessible from live PMCs. Unless, that is, we try to do something expensive like filtering or sandboxing, which would absolutely kill performance.

And this point I keep bringing up about dead objects installing references to themselves in living objects is not trivial. Our whole system is built around the premise that objects which are referenced are alive and objects which are no longer referenced can be reclaimed by GC. Throughout most of the system we dereference pointers as if they point to valid memory locations or to live PMCs. If we turn that assumption around and say that dead objects may still be referenced by the system, then we lose almost all of the benefits that our mark and sweep GC has to offer. Specifically we would either have to install tests for “liveness” around every single PMC pointer access, which would bring performance to a standstill. Otherwise, we need to have a policy that says the user at the PIR level is able to create segfaults without restriction, though officially we declare it to be a bad idea. It’s not just a matter of having to test PMCs to make sure they are alive, the memory could be reclaimed and used for some other purpose entirely! Meerly accessing a reclaimed PMC could cause problems (segfaults, etc) or, if the PMC has already been recycled into something like a transparent proxy for a network resource, send network requests to do things that you don’t want to have happen! The security implications are troubling at best.

The only real solution I can come up with to this problem, and it’s not a very good one, is to add a “purgatory” section to the GC, where we put PMCs during GC sweep but we do not actually free them. The next time GC runs, anything which is still in purgatory is clearly not referenced and can be freed immediately. Anything that is no longer in purgatory has been “resurrected” by some shenanigans and has to be treated as still being alive even though its destructor has already been called. In other words, we take a performance hit and enable zombification in order to prevent segfaults. I don’t know what we want to do here, this is probably the kind of decision best left to the architect (or tech-savvy clergy) but I just want to point out that none of our options are great.

I’ve also brought up the problem with allocating new objects during a finalizer. Why is this a problem? Keep in mind that GC tends to execute when we’ve attempted to allocate an object and have none in the free list. If we have no available headers on the free list, are already in the middle of a GC sweep and ask to allocate a new header, what do we do? Maybe we say that we invoke GC when we have only 10 items left (instead of 0) on the free list, guaranteeing that we always have a small number of headers available for finalization, though no matter what we set this limit at it’s possible we could exhaust the supply if we have many objects to finalize with complex finalizers. Every time a finalizer calls a method or boxes a string, or does any of a million other benign-sounding things PMCs get allocated. If we try to allocate a PMC when there are no PMCs on the free list and we’re already in the middle of GC sweep, the system may trigger another recursive GC run.

Another option is that we could maintain multiple pools and only sweep one at a time. If one pool is being swept we could allocate PMCs from the next pool (possibly triggering a GC sweep in that second pool and needing to recurse into a second pool, etc). Maybe we allocate headers directly from malloc while we’re in a finalizer, keep them in a list and free them immediately after the finalizer exits. We have some options here, but this is still a very ««<« HEAD:drafts/gc_destructors.md real problem that requires very careful consideration. Something like a semi-space GC algorithm might help here, because we could allocate from the “dead space” before that space was freed.

Or we could try to immediately free some PMCs during the first sweep pass, and use those headers as the free list from which to allocate during destructors. This raises some problems because it would be very difficult to identify PMCs which could be freed during the first pass without negating any references which are going to be accessed during the destructors. Also, we run into the (rare) occurance where all the PMCs swept during a particular GC run have destructors, and there are no “unused” headers to immediately free and

recycle for destructors.

real problem that requires very careful consideration. Again, I don’t have an answer here, just a long list of terrible options that need to be sorted according to the “lesser of all evils” algorithm.

3d0a502723cb7124eb717d7e82bac5ecc567ac31:_posts/2012-05-23-destructors_are_hard.md

Let’s look at destructors from another angle. Obviously a garbage-collected system is supposed to free the programmer up from having to manually manage memory (at least) and possibly other resources as well. You make a mess and don’t want to clean it yourself, the GC comes along after you and takes care of the things you don’t wnat to do yourself. On one hand the argument can be made that if you really care about a resource being cleaned in a responsible, timely manner, that you call an explicit finalizer yourself and leaving those kinds of tasks to the finalizer is akin to saying “I don’t care about that object and whatever happens, happens.” After all, if you can’t throw an exception from a destructor and if the destructor is called outside normal program flow with no opportunity to report back even the simplest of success/failure conditions, it really doesn’t matter from the standpoint of the programmer whether it succeeded or silently failed. Further, if the resource is sensitive, you don’t clean it explicitly and Parrot later crashes and segfaults because some uninformed user created a zombie PMC reference, your destructor cannot and will not get called no matter what. If all sorts of things at multiple levels can go wrong and prevent your destructor from running, does it really matter if the destructor gets called at all?

Another viewpoint is that destructors don’t need to be black-boxes, and we don’t care if they have problems so long as they’ve given a best effort to ««<« HEAD:drafts/gc_destructors.md free the resources, those efforts have a decent expected chance of success, and they have an opportunity to log problems in case somebody has a few moments to spare reading through log files. After all, if a FileHandle fails to close in an automatically-invoked destructor, it also would have failed to close in a manually-invoked one and what are you going to do about it? If the thing won’t close, it won’t close. You can either log the failure and keep going with your program (like our destructor would have done automatically) or you can raise hell and possibly terminate the program (like what could happen if an exception is thrown from a destructor). In other words, when you’re talking about failures related to basic resources at the OS level, there aren’t many good options when you’re writing programs at the Parrot level.

I suspect that what we are going to end up with is a system where we allocate a temporary managed pool of PMCs to be available, and allocate all PMCs during a destructor from that pool. After GC, we clear the emergency pool at once. This solution adds a certain amount of complexity to the GC and also does nothing to deal with the zombie references problem I’ve mentioned several times. We’d have to make a stipulation that PMCs allocated during a destructor may not themselves have automatic destructors.

Things start to get a little bit complicated no matter what path we choose. This is the kind of issue where we’re going to need lots more input,

especially from our users.

free the resources and they have an opportunity to log problems in case somebody has a few moments to spare reading through log files. After all, if a FileHandle fails to close in an automatically-invoked destructor, it also would have failed to close in a manually-invoked one and what are you going to do about it? If the thing won’t close, it won’t close. You can either log the failure and keep going with your program (like our destructor would have done automatically) or you can raise hell and possibly terminate the program (like what could happen if an exception is thrown from a destructor). In other words, when you’re talking about failures related to basic resources at the OS level, there aren’t many good options when you’re writing programs at the Parrot level. If you’re not so hot at OS administration, there might not be anything you can do no matter what.

In Parrot we really want to enable PMC destruction and GC finalization. As things stand now you can run destroy vtables written in C, usually without issue. However when we expose this functionality to the user we are talking about executing PBC, in a nested runloop (at least one!), with fresh allocations and all the capabilities of PBC at your disposal. As soon as you open that can of worms, the many problems and problematic possibilities become manifest. The security concerns become real. The performance implications become real. I’m not saying that these are problems we can’t solve, I’m only pointing out that they haven’t been solved already because they are hard problems with real trade-offs and some tough (and unpopular) decisions to be made.

3d0a502723cb7124eb717d7e82bac5ecc567ac31:_posts/2012-05-23-destructors_are_hard.md

parrot.org: GSoC starts today

brrt — 2012-05-21T11:38:44+00:00

As you may or may not know, I'm tasked with implementing mod_parrot during this years' edition of google summer of code. So in preparation, I have researched the wealth of information that is the internet, and have found the following peculiar little tool:

http://httpd.apache.org/docs/2.2/programs/apxs.html

Which stands for the apache extension tool, and which can - among other things - generate, compile, and install entire modules. I originally planned to take two weeks to do that very thing.

Andrew Whitworth: Pending Branchwork

2012-05-20T07:00:00+00:00

As I promised in my last post, I have several branches up in the air that need to be worked on. Some branches merged last week after the release. Others are pending to merge soon and some are still in development. In this post I’m going to give a short summary of these things, since I haven’t been posting regular updates like normal.

Already Merged

After the release last week I merged three small branches that brought small changes and appeared to test cleanly with NQP and Rakudo. In short, these were uncontroversial.

whiteknight/gh_675 named after the Github Issue of the same name, this branch removed the can vtable. In all cases in core and in external projects where I looked, the can vtable was simply a redirect to the find_method vtable and a check for null. There’s no need for this added indirection, we can call the find_method VTABLE directly from can opcode.
whiteknight/imcc_file_line This branch removed some very old, long-deprecated IMCC directives. The .line and .file directives were not poorly implemented (as far as IMCC goes) but they weren’t used and weren’t introspectable. The setline and setfile directives (yes, they are directives even though they looked like opcodes!) weren’t used anywhere and weren’t implemented well. I’ve removed all four. Now, we can use the .annotate directive to replace all of these and add other metadata besides in a way that is easy to introspect from within running bytecode.
whiteknight/remove_cmd_ops removed a few command-line arguments from the parrot executable which were non-functional. These arguments have been disconnected since the time of the IMCC API cleanups months ago, and nobody had even noticed. Now they’re gone.

Those things out of the way, here’s a list of some of the branches that are currently unmerged but may be merging soon.

`eval_pmc`

This is one of the most disruptive branches I’ve got going, which is why I’m in no hurry to merge it. Before I can merge it I need to patch both NQP and Rakudo. I submitted patches for these but they weren’t ready to apply and I have to go back and re-do them.

This branch removes the deprecated Eval PMC. The IMCCompiler PMC has already been updated to use a PDD31-compliant interface, which returns a PackfileView PMC instead of an Eval. NQP and Rakudo need to be updated to use this new interface instead of the older VTABLE_invoke one. This update will work in the Parrot master branch just fine, so we can make those updates to NQP and Rakudo and test them thoroughly before we merge the eval_pmc branch in.

`remove_sub_flags`

This is a much bigger and much more disruptive branch. However, because of the fact that NQP and Rakudo don’t really use subroutine flags for their control flow, those two projects won’t really be affected as much as everybody else will be.

The remove_sub_flags branch removes the :load and :init flags from the PIR syntax and replaces them with :tag. The only real way to work with :tag is through the PackfileView PMC, so we need to merge the eval_pmc branch into Parrot first before we can make any further progress on this one. This is a back-burner task and will probably not be touched before the end of the summer.

`whiteknight/gc_finalize`

We’ve received some requests from Rakudo folks that we need to start getting serious about GC finalization. This involves two changes: First is setting the GC to perform a finalization sweep at interp exit, which it currently is not doing. The second is to fix some sweep-related behaviors so the destroy VTABLE can be much more sane and useful.

The whiteknight/gc_finalize branch does both of these things. First, it re-enables GC finalization which had been turned off for so long that the code for it no longer works in master. Second, it moves to a two-stage sweep algorithm, so that we execute all destroy vtables first before we start freeing any resources.

There are still going to be problems with destroy vtables however, and I’m searching for solutions to these. Let me illustrate with a short example. We call GC to sweep typically in response to a request for a new PMC when we have none on the free list. If we have an item on the freelist, we return that immediately and very quickly. If not, we invoke GC to try and free up some headers (or allocate new ones from the OS).

Let’s say we’re programming in Rakudo Perl6 and we have an object with a destructor. For the purposes of our example, it’s a DB connection object. That destructor needs to call a method on a Socket object connecting the client program to the server. As everybody should be aware of now, calling a method in Parrot itself is going to allocate a CallContext PMC.

However, we run into a small problem because we’re in GC because we’re out of PMCs to allocate. So if we try to allocate a new PMC at this point I don’t know exactly what will happen but I can only imagine that the results would not be good. At the worst case, we recursively call into GC which goes back to sweeping, which re-executes finalizers, and we get into an infinite loop.

I won’t go into all the details here, I’ve got another (long) post drafted that discusses these and some other issues related to finalization. This whiteknight/gc_finalize branch solves some of the first few problems but there will be more to come after that.

`whiteknight/gh_663`

The singleton designator for C-level PMCs has been deprecated for some time now, and the whiteknight/gh_663 branch intends to remove them.

Here’s how singletons work in Parrot: The get_pointer and set_pointer vtables are used to manage a single reference to an existing singleton PMC if any. To get the PMC, we invoke the get_pointer vtable without an invocant PMC (the only such occurance of a vtable invoked without an existing PMC reference in the whole codebase that I am aware of). If it returns NULL, a new header is created. If the new header is created, the set_pointer vtable is called on the new object with itself as an argument.

This all happens inside Parrot_pmc_new and is mostly transparent, except for the few bits of code throughout the system which violate this (rather flimsy) encapsulation boundary.

The get_pointer and set_pointer vtables operate on void* pointers, so we even lose typesafety. Plus, we don’t expose get_pointer or set_pointer vtables to PIR code, so there’s absolutely no way to create a singleton class at the user-level using this mechanism. You can do what users of all other languages do and create an accessor and restricted constructor and implement singletons that way. In fact, I think that’s better.

The majority of offending code has been ripped out of this branch, though I’m still seeing some segfaults during the build as a result of bad, unchecked pointer accesses in places where encapsulation has been violoated. I’ve got to spend a little bit more time tracking down some of these failures. Then, assuming NQP and Rakudo aren’t relying on this mechanism, the merge should be relatively painless.

`whiteknight/gh_610`

A while ago, moritz suggested that we improve integration of our ByteBuffer PMC type, especially with our FileHandle and Socket types. We should be able to read a sequence of raw bytes from either of those PMCs into a ByteBuffer and we should be able to write raw bytes from a ByteBuffer into either of those destinations too.

The whiteknight/gh_610 aims to make this a reality. Already I’ve done most of the code work to get this in place, though I haven’t added all the necessary tests and documentation. Plus, a few coding standards tests are failing too.

While looking at this code, I am reminded that the IO subsystem is kind of messy. I’ve tried to clean it up in the past, and made a few small improvements over time. However, without a larger guiding vision to follow, I never really had a great idea of what kind of larger architectural changes to make to really bring this subsystem up out of the mud. After working on this branch, I finally had something like a flash of insight, and think I have a good idea about how to clean things up. This leads me to…

`whiteknight/io_cleanup1`

My idea is a relatively simple one: All our IO operations are controlled by the various PMC types (FileHandle, Socket, StringHandle, etc), but all our IO API functions are currently implemented as ugly (and brittle) switch statements to pick between execution pathways for these different types. A far better idea would be to separate out the different logic behind a virtual function dispatch table (vtable).

I’ve written up some proposed changes in the whiteknight/io_cleanup1 branch, and will start work if other people think it’s a decent idea.

The key points are as follows:

Move all FileHandle-specific logic into src/io/filehandle.c. Do the same for Pipe, Socket and StringHandle types.
Implement a new io_vtable type, which will contain a dispatch table for common operations. Each one of the files created in #1 above will implement the routines for one io_vtable and supporting logic.
Buffering will be refactored. Instead of the FileHandle PMC containing several attributes for buffering, we’ll instead use an io_buffer object to hold buffering details. An encapsulated buffering API will take this buffer structure and the relevant vtable and automatically perform buffering if necessary.
I am going to start separating out Pipe logic from FileHandle, though I’m not planning to create a separate type for it quite yet.

Once these things are done, I think the IO system will be much cleaner and much more hackable. This is lower priority right now until some of my ideas are vetted, but I’m glad I finally have a plan in mind after so many years of staring helplessly at this code.

`whiteknight/sprintf_cleanup`

The engine for our sprintf implementation is sort of old and messy. It’s some very functional and very stable code, but it needs to be brought up to date with our modern coding and organizational standards.

In the whiteknight/sprintf_cleanup branch I make several changes, most of which are entirely internal and should not affect users at all:

I move the files from ‘src/misc.c and src/spf_.ctosrc/string/sprintf.c and src/string/spf_.c respectively.
I’ve cleaned up some header-file nonsense and created a new src/string/spf_private.h header file to hold private data.
I’ve changed the code to use a StringBuilder instead of the older (and now-incorrect) repeated string concatenations. With immutable strings, each concat operation creates a new STRING instead of appending to the pre-allocated buffer, which is extremely wasteful. I haven’t benchmarked this change, but I suspect it has higher performance on longer, more complicated formats.
I’ve fixed a sub-optimal error message at request of benabik in ticket #759.

This branch is almost complete and I’ll probably merge it this weekend. Besides the text of the exception message, there are no visible user changes so it shouldn’t be controversial at all.

Andrew Whitworth: Parrot 4.4.0 Banana Fanna Fo Ferret

2012-05-17T07:00:00+00:00

Its existence guarantees nothing in itself, and the catalytic or Promethean moment only occurs when one individual is prepared to cease being the passive listener to such a voice and to become instead is spokesman, or representative.

But it’s important to remember the many dreary years when the prospect of victory appeared quite unattainable. On every day of those years, the “as if” pose had to be kept up, until its cumulative effect could be felt.

– Christopher Hitchens, Letters to a Young Contrarian

On behalf of the Parrot team, I’m proud to announce the 4.4.0 release of Parrot “Banana Fanna Fo Ferret”. Parrot is a virtual machine aimed at running all dynamic languages.

Parrot 4.4.0 is available on Parrot’s FTP site, or by following the download instructions. For those who want to hack on Parrot or languages that run on top of Parrot, we recommend our organization page on GitHub, or you can go directly to the official Parrot Git repo on Github.

Parrot 4.4.0 News:

- Core
    + Most internal calls to libc exit(x) have been replaced with
      Parrot_x_* API calls or PARROT_FORCE_EXIT
- Documentation
    + 'pdd31_hll.pod' made stable in 'docs/pdds/'.
    + Updated main 'README' to 'README.pod'
    + Updated various dependencies, e.g., 'lib/Parrot/Distribution.pm'.
    + Updated all 'README' files to 'README.pod' files.
    + Added 'README.pod' files to top-level directories.
- Tests
    + Update various tests to pull from new 'README.pod'
    + Updated 't/tools/install/02-install_files.t' to pull from new
      'README.pod'
- Community
- Platforms
- Tools
    + pbc_merge has been fixed to deduplicate constant strings and
      merge annotations segments

Alvis Yardley (or a delegate) will release Parrot 4.5.0, the next scheduled monthly release, on June 16th 2012. Subsequent release managers are to be announced. A special thanks to our donors, contributors and volunteers for making this release possible.

Enjoy!

I haven’t been doing enough blogging lately! On Tuesday I put out the 4.4.0 release of Parrot, “Banana Fanna Fo Ferret”. I figured it was a fun play on words. I added a little quote from a favorite writer of mine, Christopher Hitchens. Much of his writings can be pretty inflamatory, but I picked two quotes that related to historical struggles for social progress, and which when read in a certain light (and dramatically out of context) make sense for Parrot too.

The release went off without a problem, and I’ve got a few branches waiting in the environs to be merged. I’m sure I’ll talk about some of those projects if I can get back into a normal blogging rhythm again.

parrot.org: 4.4.0 Parrot "Banana Fanna Fo Ferret" Released!

Whiteknight — 2012-05-16T00:29:33+00:00

Its existence guarantees nothing in itself, and the catalytic or
Promethean moment only occurs when one individual is prepared to cease being
the passive listener to such a voice and to become instead is spokesman, or
representative.

But it's important to remember the many dreary years when the prospect of
victory appeared quite unattainable. On every day of those years, the "as if"
pose had to be kept up, until its cumulative effect could be felt.

-- Christopher Hitchens, Letters to a Young Contrarian

On behalf of the Parrot team, I'm proud to announce the 4.4.0 release of
Parrot "Banana Fanna Fo Ferret". Parrot
is a virtual machine aimed at running all dynamic languages.

parrot.org: GSOC: Introduction

jharper1 — 2012-05-14T15:14:38+00:00

Parrot Foundation,

Good day Parrot! A few people have already met me through chat, but for those who haven't, my name is Justin Harper. I am a Junior in the Electrical Engineering Department at Temple University in Philadelphia and am very excited to be apart of this team.

Andrew Whitworth: XML Is Hard

2012-04-28T07:00:00+00:00

Last week I promoted the Parse and Json libraries in Rosella to stable status. For both those libraries I wrapped up a few outstanding TODO issues, wrote up some website documentation and added a bunch of unit tests. I figured I would do the same thing for the XML library too. After all I had done the hard part: the first 90% of the library was the recursive descent parser which I had most of.

So today I got to work on that library, trying to put together the last few bits so I could make the library stable. Like I said, I had about 90% of it done already. I spent the time today doing another 90%. I figure I only have about 90% left to go before I have a “real”, usable XML library. Somewhere a mathematician is reading this post and inventing new curse words, but nobody can hear him, because he has no friends.

It turns out that XML is hard.

Anybody can put together a little parser for XML-like tag syntax with attributes, text, and nested tags. That part is dirt simple, and I had that done in an hour or two. It’s once you start getting into DTD declarations and schema validation that things get messy. Honestly, I don’t think I can seriously call Rosella’s XML library “complete” without those things. Or, not without most of them. I can probably get away with only the first 90% or so.

So, what can Rosella’s Xml library do today? Here is a sample of XML text that I can parse into a document object tree without problems:

<?xml version="1.0"?>
<!DOCTYPE foo [
    <!ELEMENT foo (bar, baz)>
    <!ELEMENT bar ANY>
    <!ELEMENT baz (fie)>
    <!ELEMENT fie EMPTY>
    <!ATTLIST fie
                lol CDATA #REQUIRED
                wat CDATA #IMPLIED
                sux CDATA #FIXED "hello!">
]>
<foo>
    <bar/>
    <baz>
        <fie lol="laughing out loud" wat="you talkin bout?" sux="hello!"/>
    </baz>
</foo>

Or, if I want, I can jam all that schema nonsense into a separate file, and load it separately:

<!DOCTYPE foo SYSTEM "foo.dtd">

Although I haven’t integrated Rosella Net yet, to allow loading schemas from a URL. In code, I can do a few things:

var dx = new Rosella.Xml.Document();
dx.read_from_file("foo.xml");
dx.validate();
if (!dx.is_valid()) {
    for (string err in dx.errors)
        say(err);
}
dx.write_to_file("newfoo.xml");

var dtd = new Rosella.Xml.DtdDocument();
dtd.read_from_file("foo.dtd");
var errors = dtd.validate_xml(dx);
if (elements(errors) > 0) {
    for (string err in errors)
        say(err);
}
dtd.write_to_file("newfoo.dtd");

That example shows us loading an XML document from a file and validating it with it’s built-in rules from the !DOCTYPE header. The second part shows us loading a separate DTD definition from a standalone file, and using that to validate the XML document too. In both cases, the validator runs through the document object and returns a whole list of error messages, not just a simple yes/no flag. In both cases, we can also re-serialize the XML and DTD documents back to string and then to file.

So what is left to do? Well, for starters there’s a bunch of syntax in the !ELEMENT tag that I don’t quite handle yet, such as quantifiers and alternations:

<!ELEMENT foo (bar*, (baz|bar), fie?)>

Parsing all that in a way that doesn’t suck is not something I’m looking forward to doing.

Then in attribute lists, there’s some syntax I don’t deal with, such as enumerated values again:

<!ATTLIST foo bar (yes|no)>

The validator I’ve implemented is pretty naive so far, and isn’t set up to do quantifiers anyway. That’s all going to take a while to do. We’re doing some basic validation now, but nowhere near as much as we would expect from a full implementation.

And keep in mind, even when I’m done implementing (mostly) proper XML and DTD parsing, I could still go on to parse other schema languages like XSD which some applications might expect and even prefer. Maybe I could do something like XPath too, which would be very nice. I probably won’t try to do XSLT though: I’m still young and I would like to keep some of my sanity in reserve for my twilight years.

My Json library is about 1300 lines of winxed code long, including whitespace. My Xml library is about 2400 lines of code long and still growing. Json is pretty easy (by design!), but XML is very hard. I’m not going to push the Xml library to become stable any time soon, there’s a hell of a lot of work left on it and I’m not going to rush anything.

parrot.org: GSoC: Here We Go Again

benabik — 2012-04-26T15:03:34+00:00

My proposal for Google's Summer of Code 2012 has been accepted!

parrot.org: LAPACK bindings for Parrot-Linear-Algebra

Jashwanth — 2012-04-25T15:50:02+00:00

LAPACK bindings for Parrot-Linear-Algebra so as to provide an interface for the basic linear algebra functions.In modern treatments of linear algebra, matrices are considered first. Matrices provide a theoretically and practically useful way of approaching many types of problems including: solution of systems of linear equations,graph theory, theory of games, computer graphics,data compression,etc.

Andrew Whitworth: Various Updates

2012-04-25T07:00:00+00:00

Here are some updates on various projects I’ve been working on or been planning to work on:

ParrotStore

In my post introducing ParrotStore, I mentioned that I only had support for MySQL, Memcached, and a little bit of stuff working for MongoDB. In the past few days I’ve also added SQLite3 support. Now you can do this, after installing the prequisites, building and installing:

var sqlite3_lib = loadlib('sqlite3_group');
var sqlite3 = new 'SQLite3DbContext';
sqlite3.open("test.sqlite3");
sqlite3.query("INSERT INTO tbl1 (name, number) VALUES ('Andrew', 100)");
var result = sqlite3.query("SELECT * FROM tbl 1");
for (var row in result) {
    for (string colname in row)
        print(colname + "=" + string(row[colname]) + " ");
    say("");
}

SQLite3 offers a bunch of features that I don’t tap into yet, but we have a good start and can do some basic work with it already.

Also, I mentioned that we didn’t support queries with multiple result sets in the MySQL bindings. Well, now we do (and we do in SQLite3 too):

var result1 = mysql.query("CALL my_stored_proc");
var result2 = sqlite3.query("SELECT * FROM tbl1 ; SELECT * from tbl2");

If the query returns one result set, a DataTable object is returned. If it has multple result sets, an array of DataTables is returned instead.

Eval PMC

I went digging through my backlog of old branches last night and found my incomplete branch for removing the deprecated Eval PMC. After updating to current master I gave it a spin and most things looked good. I fixed all the core parrot tests and then moved on to the rest of the ecosystem.

Winxed works fine with the PackfileView PMC instead of the Eval PMC. I made a few of those updates in the past, so it mostly worked out of the gate. Rosella compiled and ran like a charm too.

NQP-rx works fine because it mostly relies on the PCT libraries that ship with Parrot, and which I had already fixed.

The new NQP is a little bit more of a hassle. It took me a little bit of effort to figure out the bootstrapping mechanism, but after a few hours of hacking I had NQP building on the new Parrot using PackfileView instead of Eval. However, one of the regex tests hangs indefinitely now and I’m having trouble tracking that down. this project may get bumped down to a lower priority level until I can either figure out what the problem with NQP is, or until I can enlist some help to fix it.

I would like to merge this branch as soon as NQP is fixed and I can prove that I can build it and Rakudo on the branch.

Sub Flags Cleanup

My remove_sub_flags branch, tasked with removing the old :load and :init flags from Parrot and replacing them with the new :tag() syntax is right where I left it a few weeks ago. I’m down to a relatively small list of test failures, the solution to most of which is to update the syntax in the tests themselves. A handful of tests such as those using the parrot-nqp and winxed compilers are failing because I need to update those compilers first to generate the correct code so the tests can run correctly.

After fixing NQP-rx and Winxed, I need to get started testing out the new NQP and Rakudo. I suspect both of those two things will be made to work without too much effort.

It turns out that the Eval PMC deprecation work overlaps with this slightly, so the things I change for that branch should help reduce failures in this branch too. After I get Eval deprecated and removed, I’ll come back to this branch and see where things stand.

This is such a large and disruptive change that I can’t imagine we would want a merge before the 4.4 release, even if I got all the bugs ironed out. We could be a month or more away from a merge, so I’m not listing this work as high priority.

PCC

Bacek has been doing a lot of refactoring in PCC land, trying to fix some slow and infelicitious aspects of it. I’ve gotten a set of new PCC-related opcodes added to core and have a few more that I want to add, including new variants of set_args, get_params and friends to take explicit context arguments instead of using magical behavior to try and find them automatically. A few patches to IMCC and the new behavior might go in without anybody noticing. I’ve talked more about this in past posts, and I’m sure I’ll have more to say when I start making changes.

Rosella

Rosella is mostly where I want it to be right now. I’m planning to change around the development cycle to stick to supported releases of Parrot and Winxed instead of tracking HEAD for both of them. I’m going to promote one or two more libraries to “stable” status and then put out a release sometime after Parrot 4.4 hits the news stands next month. I’ve already promoted the Parse and Json libraries to stable status. I will probably promote Xml and Net too, since I am pretty happy with both of those two libraries and feel that they are almost ready for general use.

After that, I suspect Rosella is going to take a back seat for a while, so I can focus on some other projects.

Google Summer of Code

GSOC is keeping me pretty busy so far. We accepted 4 projects this summer. The fifth project, which was to do some work on the Jaesop Stage 1 compiler, was lost because the student was accepted to a different organization instead. The four remaining projects are:

Security Sandbox by Justin
Mod_Parrot 2.0 by brrt
LAPACK Bindings by jashwanth
PACT Assembly by benabik

I think these projects will be very cool, and I am looking forward to see what kinds of great code they can produce this summer.

Green Threads

nine has been doing some amazing work on his threading branch. Yesterday he informed me that he had a solution to make green threads work on Windows, and had already implemented part of it. That’s awesome, because I was planning to work on porting the green threads to windows next, but if he’s doing it then I don’t have to.

Some of the performance numbers he’s been getting are pretty impressive for certain tasks. Some benchmarks he has are even showing a significant threading performance improvement over a similar benchmark written in perl5.

I’ve been doing some testing on his branch and things are looking mostly good except for one or two remaining GC-related bugs that need to be ironed out. After that, if we can get some concensus, I would love to start talking a merger shortly after 4.4.

6Model

With Green Threads possibly off my TO-DO list, Eval PMC Deprecation mostly wrapped up and remove_sub_flags on the back burner, I can start moving towards my next project: 6model. And I can do it much earlier than I was expecting. I’m going to mine benabik’s rejected 6model project proposal for some ideas, then I’m going to jump in and try to get things working. I suspect things could get moving pretty quickly, if I can keep my level of free time relatiely high.

parrot.org: Hello Parrot

brrt — 2012-04-24T20:44:02+00:00

Hello everybody,
Let me introduce myself. I'm Bart Wiegmans, brrt on irc, and I will be implementing a new mod_parrot this summer, courtesy of google and whiteknight. Which will make it possible to serve parrot-powered web applications once again.

So, who am I? I'm 24 years old and I study biology in Groningen, the Netherlands where I live together with my girlfriend. Aside from that, I work as a programmer for studenten.net. As for hobbies, I play mario kart and pikmin. I used to say I had four pet rats, but unfortunately one of them died just Last weekend.

parrot.org: Test Post

brrt — 2012-04-24T20:18:50+00:00

This is a test post.
Lorem ipsum solor dit amet.

Tadeusz Sośnierz (tadzik): Back from Oslo

ttjjss — 2012-04-23T18:30:08+00:00

And so the hackathon is over. I haven’t experienced anything this awesome since I discovered how well bacon tastes with maasdamer.

I seem to be much more productive during hackathons if I have no talk to prepare. I’ve had lots of plans about it (see previous post), and while I haven’t started even half of them I think, I’m really satisfied with what was achieved.

On friday I looked a little bit into how Rakudo loads precompiled modules and why doesn’t it always work as expected. The easiest one to notice was Test.pm, which for some unknown reason was installed only as a source file, not a precompiled module. A simple fix in the Makefile, and suddenly every test file you ever run runs about 3.5 seconds faster. Suddenly it makes module development a lot less painful.

The other thing about module loading was that a lib/ directory was always at the very beginning of the @*INC array – that means that precompiling your modules (which usually puts them in blib/) before running your tests gave you… right. Absolutely nothing. How significant is that? Well, tests of Panda take just under 7 seconds when you have the modules precompiled, and almost 23 seconds when they aren’t. So that’s about 3 times faster test runs for your modules. How cool is that?

That solved pretty much all the issues and wtf’s I’ve experienced with module precompilation. It took me another commit on saturday to teach panda to precompile modules again. Then my mind wandered again around the idea of emmentaler, a service for automatic smoke testing of Perl 6 modules. A quick and dirty 100-lines long script is now able to download all the modules, test them and present the results in a bit silly, but definetely useful way. There’s still lots of things to be done about it, but the fact that you can test all the modules in about 20 minutes is something that wasn’t quite possible before. Huge success; I hope I can turn it into something that the whole community will benefit.

Saturday also brought a couple of fixes to Pod handling and the –doc command line switch. After a quick discussion with Damian Conway, sort of a “what does the spec really expect me to do”, I taught –doc to Do The Right Thing whatever argument you pass to it. So –doc=HTML will use Pod::To::HTML for Pod formatting, and whatever module you write is available to be used this way without changes neither in Rakudo nor in the module code itself. A small thing, but something that waited much too long to be done after GSoC. It should now be possible to extract documentation from modules in a simple, automatic way, possibly for inclusion on modules.perl6.org or somewhere? We’ll see about that.

The same day Marcus Ramberg worked on Pod::To::Text, in particular the .WHY handling part. He also contributed some patches to Rakudo itself, and thanks to him what –doc prints for documented subs and methods is actually Something Useful and not Something That’s Good That It Exists. Much appreciated!

Sunday brought some new fixes to Bailador::Test, which now does almost everything that Dancer::Test does, so some brave soul could actually start porting tests from Dancer to Bailador to see what blows up. I’ve also spent plenty of time (mostly compilation time) chasing small things like a few-months-old panda bug, which actually turned out to be a Rakudobug, making perl6 –doc behave a bit more like perldoc, and less like “what the hell does this beast do”, and some other small things.

Besides hacking there was also plenty of opportunities to walk around Oslo, try some delicious food and beer and meet people mostly known from IRC before. Social aspect of the Hackathon was probably the best part of it, even given the fact that the coding aspect was probably the best one I’ve ever experienced. This wouldn’t have been possible without the work that the organizers have put into the event. It was absolutely perfect. Thank you, Oslo.pm! And thank you, sjn for giving me the opportunity to be there. It was a great hackathon and a time well spent. I hope to see you guys sooner than later. And if later than sooner, let it be the next hackathon in Oslo, for there’s no way that I’ll let this one happen without me.

parrot.org: Parrot 4.3.0 "In Which..." Released!

cotto — 2012-04-18T06:43:22+00:00

On behalf of the Parrot team, I'm proud to announce Parrot 4.3.0, also known as "In Which...". Parrot is a virtual machine aimed at running all dynamic languages.

Andrew Whitworth: ParrotStore

2012-04-15T07:00:00+00:00

I created a new repo for a new project: ParrotStore. ParrotStore intends to provide some storage and persistance (and caching and database) solutions for Parrot. At the time of writing this post we have three in development: Memcached, MySQL and MongoDB.

Memcached

The first thing I wrote is a rudimentary pure-parrot interface to Memcached for high speed caching. The interface looks like this:

var memcached = new ParrotStore.Memcached(["192.168.1.1", "192.169.1.2"]);
memcached.set("foo", "hello world!");
:(int have, string content) = memcached.get("foo");
say(content);

Or, if you want a simpler interface, you can do something like this:

string content = memcached.autoget("foo",
    function() { return "hello world!"; }
);

The autoget method will try to read from Memcached if the item exists, and will invoke the callback to get the value otherwise (and save it to Memcached for later use). Of course, for this to be practical the callback to generate the content should be more expensive than a return of a constant string.

I havent’t tested with multiple memcached servers yet, and I haven’t implemented several of the methods memcached supports. It’s a start, however, and I can already think of several potential uses for it.

MySQL

MySQL is popular and extremely common, so I figured I should work on that next. Plus, if we ever want to have a snowball’s chance in hell of hosting a decent PHP compiler, we’re going to want easy and available bindings for MySQL. Now, after a little bit of hacking today, we have it.

Here’s what we can do in Parrot today:

var lib = loadlib("mysql_group");
var mysql = new 'MySQLDbContext';
mysql.connect("localhost", "username", "password", "database", 0, 0);
var result = mysql.query("DROP DATABASE foo;");
say(result, " rows effected");      // "1 rows affected", if you had one

result = mysql.query("SELECT * FROM bar");
say(typeof(result));                // "MySqlDataTable"
for (var row in result) {           // Iterate over all rows
    int idx = int(row);
    say("row " + string(idx));
    for (string column in row) {    // Iterate over all columns
        say(column + ": " + string(row[column]));
    }
}

One thing I don’t handle quite yet is handling multiple result sets. So if you have a stored proc which returns multiple sets of data, you won’t get any but the first back into your program. I’ll try to get that implemented as quickly as I can.

MongoDB

We’re starting to use MongoDB at work, and I figured a great way to become more familiar with this piece of software was to write bindings for it for Parrot. Despite several unnecessary problems with linking to the Mongo C Driver libraries, I’ve managed to produce a few results.

Mongo uses a storage format called BSON (similar to JSON), and stores BSON documents as atomic units. ParrotStore implements a BsonDocument and a MongoDbContext PMC type. As of this morning, you can create a BSON document and insert it into the DB:

var lib = loadlib("mongodb_group");
var bsondoc = new 'BsonDocument';
bsondoc.append_start_object("name");
bsondoc.append_string("first", "Andrew");
bsondoc.append_string("nick", "Whiteknight");
bsondoc.append_end_obect();
bsondoc.finish();

var mongo = new 'MongoDbContext';
mongo.connect("127.0.0.1", 27017);
mongo.insert("local.foo", bsondoc);

The document is indeed written to the database, although I don’t have any methods yet to read it back out. The documentation for the C Driver for MongoDB is lacking, but I have the source code handy and it is pretty readable. I hope to have basic querying implemented by the end of the day.

Here are a few things I plan to add, either today or in the next few days:

Support simple querys and commands
Support introspecting and iterating over BSON documents
Implement a JSON->BSON translator (I have most of this written already).

There are several other features that I need to implement, although many of them aren’t necessary to say I have a minimally functional set: support for replicated sets, support for atomic find/replace updates, support for cursors and bson iterators, etc. There’s a lot of work here, but I’m off to a pretty good start already.

Build System and Project Setup

ParrotStore contains a bunch of sub-projects which are really only related by theme. They’re all solutions for storing stuff, but they don’t really relate to each other besides that. So, the build system is set up to easily build these projects individually. At the terminal, if you have make, you can build them like this:

make memcached
make install_memcached
make mysql
make install_mysql
make mongodb
make install_mongodb
make            # attempts to build them all
make install    # attempts to build and install them all

This is great for if you don’t have the mysql or mongodb development packages installed but you want to get the memcached library (or any other combination).

Internally, the makefile calls a distutils-based setup.winxed program for building the various components, but you shouldn’t use setup.winxed directly.

Like Rosella, which is a prerequisite for this project, ParrotStore will be a collection of things not one big monolithic system. It will provide a Memcached interface in one standalone library, a MySQL interface in one, a MongoDB interface in one, and other interfaces separately too. Some of them (like Memcached) will be pure parrot. Other things like MongoDB will have C-level components too. Where Rosella has always promised to be pure Parrot, ParrotStore cannot and should not follow such a rule. Some things may turn out to be implementable with NCI, but that’s an experiment for later. Maybe, much later.

Also, expect a lot of synergy between Rosella and ParrotStore. ParrotStore will both use Rosella internally, provide many of the interfaces that other Rosella-based projects expect, and add several extensions to make Rosella features even more cool and powerful.

Future Projects

The goal of ParrotStore is simple persistance. In a sense it might become something like an ORM, or contain an ORM, mapping Parrot data to and from various persistance mechanisms. This project does not intend to do any embedding, whether Parrot embedded in a database or a database embedded in Parrot, or whatever else. The Database (or cache or whatever) is separate, and ParrotStore just provides a client interface to it. For instance, the PL/Parrot project embeds Parrot into the Postgres DB. ParrotStore would provide an external interface for querying it instead.

I do not yet have a runnable test suite. I’ve been doing ad hoc tests because this is all so new and experimental. I need to add a test suite.

I also want to add a custom caching mechanism for storing frozen PMCs to file and fetching them again. Multiple backends to a PMC mechanism would allow us to store PMCs to various persistance systems for later use. This is another thing that I’ve wanted for a while, but I haven’t quite nailed down a design yet.

I would like to add a client interface for Postgres. I suspect there are some people floating around who could help make that a reality.

I think this project will probably grow organically, adding new storage backends and cool interfaces for various purposes, and then adding some tools and utilities that use these things. As with all my projects, feedback, requests, suggestions, and questions about my basic compentency are always welcome.

Tadeusz Sośnierz (tadzik): Plans for the Perl 6 Hackathon in Oslo

ttjjss — 2012-04-10T20:20:10+00:00

The Perl 6 Patterns hackthon in Oslo is happening next week, gathering most of the Rakudo developers. Awesome!

I’m still looking for the thing I’d like to work on during the event. The first obvious thought is Bailador, the Perl 6 port of Dancer web framework. It’s getting more and more in shape, and I’ve recently started looking through the code of Dancer 2 to see how it all should be done. I really don’t want Bailador to share a fate of neutro (ie reaching a state of being a horrible, unmaintainable mess), and I feel that it’s time for it to be refactored a bit. It came to life as a 10-line module written on a piece of paper during my Spanish classes, and simply adding more and more features to it is not going to end well.

Moritz Lenz is planning to hack on database stuff, which combined with a useful and usable web framework could create a nice platform for web development in Perl 6. I’ve already heard people saying “well, I could write Perl 6 apps if it had a web framework and some database access”, so I guess it’s time to show off something suitable for web development, even simple, but usable and covering most usage scenarios.

While we’re at the Web-y stuff, one thing that we really miss is a good (or at least good-enough) templating engine. Bailador has been using Ratel for some time, yet for one it stopped working on Rakudo recently (it was probably relying on some buggy or incorrect behaviour), and it suffers the problem of having no maintainer (no one really claims ownership of it). Hitomi was to be a templating engine for the new age, so maybe it’s about time to bring it back to life in all its glory? One could definitely find something to hack on.

From the new things, but still related to web development, zby has proposed porting WebNano to Perl 6 as one of the hackathon projects. It’s not something very big, and it should be totally possible to rewrite it in Perl 6 to run on Rakudo.

Or maybe there is something you particulary want to see in Perl 6? I’m sure we’ll not lack people with tuits; what is there that could possibly make Perl 6 a useful tool for you, something that you were missing when you last looked at it?

Andrew Whitworth: GSOC Proposals Received

2012-04-09T07:00:00+00:00

The GSOC 2012 proposal deadline has come and gone. We’ve received several project proposals, although only half a dozen are serious, honest, plausible proposals. We have a few days now to rank them, comment on them, and assign potential mentors to them. When we find out how many slots Google is able to assign to us, we’ll be able to pick out which ones will be worked on this summer.

Here’s a list of the decent-looking proposals we’ve received:

Jaesop Stage 1 Compiler and Runtime by mayank. mayank intends to fix up the last remaining bits of stage 0, then get started on a Javascript-compiler-in-Javascript stage 1. By the end of the summer I think it’s very plausible that he could have a self-hosting compiler for most of the JavaScript language and at least a start on the basic runtime. If he keeps the abstraction boundaries nice and clean, after the summer is over we should be well primed to start upgrading bits of the internals to use 6model and PACT, when those two projects are ready to be used there.
LAPACK Bindings for Parrot-Linear-Algebra by jashwanth. This is something I’ve wanted to add to PLA since the beginning of that project, and an absolute necessity if I ever want to get back to my dream of writing an M language compiler for Parrot. jashwanth has proposed assing LAPACK bindings to PLA (via NCI) and implementing a nice interface for some of the most important transformations, decompositions and operations provided by that library. He also intends to provide a few pure-parrot backup implementations for cases when LAPACK isn’t available but we still need to get work done. It’s an open-ended project that can be done in small, discrete chunks.
6model Integration by benabik. We know we want 6model, and we know benabik has the chops to pull it off. He is still working on his thesis AND is expecting a baby this summer, but somehow I still don’t feel like it’s an undoable project. His proposal is to integrate 6model into Parrot’s core and start transitioning our existing PMCs to use 6model instead (and abandon most of our current object model).
PACT Assembly by benabik. Yes, benabik has submitted two proposals. This one is the start of PACT; something that, like 6model, we know we want. benabik, considering PACT was his brain child and he’s getting his PhD in compiler-related topics, is uniquely qualified to pull this one off and make it shine. The real question is which one of his two proposals we as a community want him to work on more (I’m already personally signed up to do whichever one he doesn’t pick, so it shouldn’t be a loss either way). PACT, as I’ve talked about before, is intended to be a large modular library of compiler tools and building blocks, so there is ample room to expand the project if things are going unusually well.
Security Sandbox by Justin. Security sandboxing is something that we’ve wanted, to varying degrees, for years. Justin has proposed to at least get us started with proper security and implement as many permissions and restrictions as he has time for. It’s a project that we can consider to be a “success” if even half of what gets proposed actually gets completed, and there is plenty of room to build on if his momentum gets up.
Mod_Parrot 2.0 by brrt. ModParrot, the Parrot module for the Apache webserver hasn’t been actively maintained in some time, and has fallen into disrepair following many of the internals changes to Parrot in the past few years. brrt has proposed an update to ModParrot to use the new and more stable embedding API. This is another modular project that can grow if his development speed stays high to include all sorts of helper libraries, driver programs, plugins for HLLs (Rakudo in particular) and other things. Most valuable at all may be his plans for implementing an automated test suite, which will help ensure ModParrot never falls by the wayside again.

So we’ve gotten 6 decent proposals from 5 students, and if even half of these go on to succeed in reaching their goals Parrot will be much better off by the end of the summer. And this list doesn’t even include the calling-conventions work that bacek is working on, or the threading work that nine is working on, The M0 work that several other developers are doing, or the packfile and IMCC and whatever else work that I’m planning for myself. This could be a very eventful summer indeed.

If you’re signed up to be a mentor this summer, or if you would like to be, please head over to the GSOC website, sign up, and take a look at the proposals.

Tadeusz Sośnierz (tadzik): Asynchronous HTTP requests in Perl 6

ttjjss — 2012-04-02T15:38:43+00:00

Inspired by http://oylenshpeegul.typepad.com/blog/2012/03/asynchronicity.html, I decided to tackle asynchronous HTTP requests in Perl 6 on Rakudo. Allowing this required a bit of hacking on MuEvent, and the changes are far too ugly and hacky to be released, but here’s how it looks:

    use MuEvent;

    my @urls = <
        cpan.org
        kosciol-spaghetti.pl
        perlcabal.org
        duckduckgo.com
        perl6.org
    >;
    my $count = @urls.elems;
    my $starttime;

    sub handler ($what, $content) {
        say sprintf "%-25s has loaded in %s", $what, now - $starttime;
        unless --$count {
            say "Finished in {now - $starttime} seconds";
            exit 0;
        }
    }

    for @urls -> $url {
        http_get(url => $url, cb => sub { handler($url, $^content) })
    }

    $starttime = now;

    MuEvent::run;

The code results in something like this:

    cpan.org                  has loaded in 0.047303409195493
    perlcabal.org             has loaded in 0.0985883954326499
    duckduckgo.com            has loaded in 0.137316369015216
    perl6.org                 has loaded in 0.175586713955562
    kosciol-spaghetti.pl      has loaded in 0.413971411974607
    Finished in 0.465130828044337 seconds

You may notice that the urls are pretty much ordered – that’s because my internet connection is probably way faster than MuEvent’s event loop :)

For now http_get() is using bare sockets to send HTTP requests, which is, well, Less Than Awesome at least. Once I figure out how to properly tie LWP::Simple to MuEvent I’ll hopefully hack out some proper solution to be released for everyone’s joy. Stay tuned.

Andrew Whitworth: Jaesop Modules

2012-04-01T07:00:00+00:00

I’ve received a few emails from prospective GSOC students interested in doing Jaesop-related work this summer for GSOC. So, to make sure it was a platform that’s fit to be worked on, I fired it up and ran through the tests. Everything passed, which is pretty awesome, except there weren’t a whole heck of a lot of tests to begin with. Saying that 100% of tests covering about 10% of the code passed isn’t saying much with any kind of certainty.

Having a student work on Jaesop, if such a proposal is submitted and accepted, would be quite a boon for that project. In summers past we’ve had students working on compiler-related projects, usually starting with nothing or almost nothing. For instance last year Rohit was working on a JavaScript compiler starting from the ground up and didn’t have a huge amount of luck. Lucian was working on a Python compiler project, disregarding much of the older Pynie project work that had been done. He did better, but would have been able to achieve a lot more if he were starting from a stronger foundation (especially, an improved Python-ready object model). Asking another student to work on a new compiler project this year would not be a great thing for us to do, especially knowing that some of the fundamental issues (i.e. object model) are still not resolved at the lowest level.

Jaesop is slightly different because a student would be starting with a working foundation. It’s not perfect by any stretch, but it is something. It is a working piece of code with some of the complexities of the JavaScript object model and library logic already sorted out. To make it an even better platform for launching a summer project, I decided that a few more pieces needed to be added. I feel like it’s the difference between a student who can hit the ground running and one who has to crawl around a few big rocks first.

First thing, I added require() and exports. This way we can do the Common.js analog of loading bytecode files. Here’s a small example that I wrote out, called sys.js:

exports.puts = function(s) {
    WX->say(s);
};

And here is how we would call that from our program:

var sys = require("sys");
sys.puts("Hello world!");

Doing this kind of stuff is the kind of unnecessary complexity that a student shouldn’t have to worry about. It’s tangential to any problems a student would be solving and so having the student waste time on infrastructure like this would just be taking time away from the actual project.

I’ve also improved logic relating to protypes. It’s not perfect or compliant by any standards, but it’s much much closer to the norm and likely gets us close enough to parsing and running the kinds of non-trivial programs that are going to form the basis of a compiler.

var sys = require("sys");
function Foo() {
    this.a = "foo a";
}
Foo.prototype.b = "foo_b";

function Bar() {
    this.a = "bar a";
}
Bar.prototype.b = "bar b";

var f = new Foo();
sys.puts(f.a);
sys.puts(f.b);

var b = new Bar();
sys.puts(b.a);
sys.puts(b.b);

If I didn’t tell you otherwise, you might think this was normal JavaScript code running on Node.js or some other real JavaScript compiler! This exact code example is running on my machine right now. Prior to my hacking this morning, the code above would either have thrown an error or have printed out the same thing twice because both Foo and Bar function objects would have shared a prototype.

Now that I’ve done this work I feel much better about a Jaesop-based GSOC project happening this summer. Now the responsibility lies with the students to submit acceptable proposals and get to work!

Andrew Whitworth: Rosella Json and Xml

2012-03-31T07:00:00+00:00

In a previous post I wrote about how I didn’t have XML or JSON parsing libraries in Rosella and didn’t have any real plans to build them either. Well, I kind of lied. As of this morning, Rosella has an experimental prototype of an XML parsing library and a JSON library.

On the XML side, you can do something like this:

var d = new Rosella.Xml.Document();
d.read_from_string("<foo><bar value='hello!'/></foo>");
say(d.get_root_element().children[0].attributes["value"]);
d.write_to_file("test.xml");

For JSON you can do similar things:

var obj = Rosella.Json.parse("{ 'foo':3.14, 'bar': [ null, []] }");
string json = Rosella.Json.to_json(obj);

I was looking at the ByteBuffer and StringIterator types, which both allow relatively quick access to individual characters as integer values. Originally I didn’t want to make an XML or JSON parsing library because I figured that string/substring operations to break the string up would be far too slow. However, if you iterate the strings as integers and make ample use of Winxed’s compile-time constants and code inlining abilities, the performance can suddenly become comparable to any other data parser in other similar languages.

Things could actually be faster still if I could assume all strings are ASCII. However real-world data doesn’t play by those rules. So, we need to go through the added layer of abstraction in StringIterator to get characters, which might not be fixed-width depending on encoding. Things could also be faster if StringIterator had a fast “unget” or “roll back” operation. Instead I need to maintain a relatively costly character stack for parsing, which is not quite ideal.

Are these libraries standards compliant? No. Does the XML library know anything about XSLT, XPath, SAX, or schemas? No. Do these libraries handle all error conditions gracefully? No. Are they well tested right now? Emphatic no.

Parrot’s standard library comes with a data_json compiler object which can compile JSON to an object, or serialize an object back to JSON. I haven’t benchmarked my implementation against that one, but I have many reasons to believe that mine is significantly faster. It’s still immature and probably very error prone, but it’s only existed for a day and there are many improvements left to make.

These are basic and immature implementations of data format parsing libraries, and excellent proofs of concept. I need to add plenty of tests for each and then start benchmarking to make sure the bit-twiddly effort I’ve put in to creating quick algorithms has paid off. Rosella has two new toys to play with, and I hope they can do some good in the coming months.

Andrew Whitworth: Rosella Net

2012-03-25T07:00:00+00:00

I’ve started work on a new library for Rosella: Net. Net will be a networking library for doing things like HTTP requests and related operations. For a first draft I’m modelling it closely on the LWP::, Http::, and related modules in Parrot’s standard library. Many of these have been lovingly crafted and maintained by Francois Perrad and others over the years. Basically I’m borrowing the essential algorithms from the existing libraries, translating to Winxed en passant, and using them as a basis for building a bigger library.

I’m not looking to replace the LWP or Http modules from Parrot. Those things are required dependencies of Distutils, which is used by many projects for good reason. Distutils is an amazing tool for quickly and easily setting up build and test frameworks for Parrot-based projects. I wouldn’t want to disrupt that. In fact, I want to encourage it. Rosella uses Distutils for its build. PLA uses it as well. It’s a great tool and most projects on Parrot should consider using it. What I do want to do is create an internet-access framework where basic functionality is easy to get to and where new features can be easily added. I also want to fill in what I see as being a pretty important functionality gap in Rosella’s collection of libraries.

The LWP and Http libraries in the Parrot standard runtime are loosly-based on the Perl 5 modules of the same names. They started as something of a direct translation of the necessary parts of the Perl5 libraries directly to PIR. What I want to do is start with some of the ideas and algorithms from that port, update the interfaces to make them more Rosella-esque, and then start adding some of the features that the LWP and Http ports didn’t include (and maybe more beyond that).

Eventually I would like to add a nice wrapper interface around Socket and Select PMCs the same way the FileSystem library adds nicer wrappers around FileHandle and OS PMCs. I also want to add support for other protocols (FTP and IRC both come to mind). SOAP, REST, and RPC calls could be interesting to add in the future, though most of those would require a library for building and parsing XML, which I don’t have now and don’t have any plans to add in the near future. If I could add upload support for Smolder or other continuous integration applications to my Harness library, that would be fun too.

As of today I can upload report archives to smolder which have been previously assembled by distutils:

var request = Rosella.Net.Http.create_request("http://smolder.parrot.org/app/projects/process_add_report/2");
request.set_method("POST");
request.add_header("Connection", "close");
request.add_form_field("architecture", "x86-64");
request.add_form_field("platform", "linux");
request.add_form_field("revision", "0e75eb61d5b713b57b05e91a2d45251bb18d3e2e");
request.add_form_field("tags", "test");
request.add_form_field_filename("report_file", "/pla/report.tar.gz", "application/octet-stream");
var response = Rosella.Net.Http.request(request);
say(response.status_code());
say(response.header().get_header_text());
say(response.get_content());

This is a lower-level of interface than I expect most people to use when the library is more mature. Eventually you’ll create a UserAgent object which provides nice pretty methods for making GET and POST requests. Like I said, this is roughly based on Parrot’s LWP library, which itself is based on Perl 5’s much-loved LWP module.

The library also supports local resource access with file:// URIs:

// Get a file:
var request = Rosella.Net.Http.create_request("file:///home/andrew/data/foo.txt");
var response = Rosella.Net.Http.request(request);
say(response.get_content());

// Create a new file:
var request = Rosella.Net.Http.create_request("file:///home/andrew/data/bar.txt");
request.set_method("PUT");
request.set_content("Hello World!");
var response = Rosella.Net.Http.request(request);
say(response.status_code());

There isn’t a lot to look at with this new library quite yet, but things are moving along pretty quick. Now that I have some of the basic algorithms working, I’ll start rearranging the internals to more closely align with my long-term goals. This has basically been a weekend project so far, and I need to get back to my remove_sub_flags branch and a few other projects in Parrot core before I take this too much further.

parrot.org: Parrot 4.2.0 "Ornithopter" Released!

dukeleto — 2012-03-21T12:21:41+00:00

"Technology, in common with many other activities, tends toward avoidance of
risks by investors. Uncertainty is ruled out if possible. Capital investment
follows this rule, since people generally prefer the predictable. Few recognize
how destructive this can be, how it imposes severe limits on variability and
thus makes whole populations fatally vulnerable to the shocking ways our
universe can throw the dice."
-- Assessment of Ix, Bene Gesserit Archives (Heretics of Dune)

On behalf of the Parrot team, I'm proud to announce Parrot 4.2.0, also known

Andrew Whitworth: Parrot In GSOC 2012

2012-03-16T07:00:00+00:00

I just got the email a few minutes ago: The Parrot Foundation has been accepted into the Google Summer of Code program for the summer of 2012!

I haven’t wanted to write too much about GSOC until I knew we were accepted. I thought (hoped) that our application and track record were strong, but I don’t like to take anything for granted. But, now that we are accepted I can (and will) write as much about it as I want. Unfortuanately I don’t have time to write much today, so here are some links that prospective students should read over until I have the time to spend in front of my keyboard:

In the coming days I’ll be writing about more project ideas.

Andrew Whitworth: Packfile Write API

2012-03-15T07:00:00+00:00

Last night benabik told me about a problem that, while serious, hardly caused me to raise an eyebrow. Some innocuous-looking code, written while trying to follow along with some ill-written documentation, lead IMCC to enter into an infinite loop. I wasn’t surprised that IMCC contained such a bug, though I was surprised that it was so easy to reproduce and the functionality in question wasn’t really tested at all. The code that caused this bug is this:

.const 'FixedIntegerArray' foo = 'test'

The Sub ‘test’ was an :immediate subroutine which generated a FixedIntegerArray. For people not familiar with IMCC’s workings an :immediate Sub is executed as soon as the packfile is compiled and its entry in the packfile is replaced with the return value of the Sub. So, the Sub doesn’t exist in the packfile, only the thing that the Sub generated. This is the mechanism by which arbitrary PMCs can (in theory) be serialized into the packfile at runtime.

Compare with this syntax:

.const 'Sub' foo = 'test'

In this example, the local variable ‘foo’ will refer to the item in the constants table where the Sub ‘test’ is (or would be, if it weren’t an :immediate). The tag ‘Sub’ there is a bit of a misnomer, since the PMC in that slot might not be a Sub. This syntax is basically a really lousy way of saying “give me the PMC in the constant table in the slot where the given named ‘Sub’ is or would be.” Also, that type information isn’t really used for anything, since PIR is a dynamic language. At first glance you might suspect that the .const directive can take any PMC name as its type, but that is wrong. It only accepts four types: 'Sub', 'Integer', 'String' and 'FixedIntegerArray'. Don’t ask me why these are the only four supported from among all our built-in types, or why we support more than just ‘Sub’ (the other three options seem superfluous to me).

The poorness of this syntax is not really something that I want to write about in this post. We can do it better and in the next incarnation of PIR (if we ever get to building such a beast) will do better. This is just one more “of course it does it that way” kind of moment that you learn to ignore when you’re dealing with the code that is the current incarnation of IMCC.

What I do want to write about instead is the file where the parsing code for this exists: compilers/imcc/pbc.c. That file contains much of the logic for taking IMCC’s internal SymReg representation and turning it into a packfile. It’s a huge mess, and encapsulation is broken here in ways that are as bad or worse than any other single example in the repository. That’s why I’m planning a shotgun cleanup of it soon.

The functions and logical blocks in this file fall into two broad categories: First, there are functions for iterating the AST (struct SymReg and friends) and pulling out relevant values. Second, there are functions for inserting new data into the budding packfile. The first category of functions is generally fine and a necessary part of any assembler, even if some of the code could be cleaned and modernized. It’s that second category that’s of much broader interest. My plan is to take functions and sequences from pbc.c out of IMCC, wrap them up all pretty, and add them to the proper packfile subsystem API.

Of course, I do start to think about how exactly to do that. At runtime the PackFile* structure is basically read-only. Bytecode is read-only and contains fixed integer indices into the constants table which is also not expected to change. If bytecode isn’t changing then annotations and debugging info for that bytecode is probably not changing either. Once a packfile is loaded into the interp, given a PackfileView PMC wrapper, and made executable it really shouldn’t be modified any more.

However when we’re talking about a compiler or other code generating system, we want the ability to write and modify packfiles. When we’re done modifying we might want to stamp them with a flag to say that they are read-only and suitable for executation.

So I’m thinking we want two APIs. The first uses a bit flag on the packfile to determine if it’s editable, and can edit it if that flag is set. The second is the normal read-only accessor API which can generally ignore that flag except for the routines that load a packfile in to be executed by the interpreter. For those handful of routines that do actual loading and verification we can throw an exception or something.

My general plan, and I want to get a lot of feedback on this before I touch anything, is to make a second API for packfile editing routines. I’ll prefix those functions with something like Parrot_pfw_ (instead of Parrot_pf_) to set them aside. I’ll then start moving packfile building logic from IMCC into this new API. Instantly we get much improved encapsulation, clear separation of concerns between packfile writing and executing, and a more robust interface for compiler writers to use in the future. It will also be a nice development in terms of security, where we can limit certain packfiles to be executable. I think it’s a pretty good idea and I don’t think it should take too much effort to accomplish. At least, not in a first draft.

I’m not starting any new projects until I get my remove_sub_flags branch much further along. I think it’s a good idea to follow up a tough project with one which is more straight-forward and offers clear rewards. Speaking of that branch, parrot is building fine and I’m slowly working my way through the list of test failures. When I get the majority of those sorted out I’m going to start working on patches for Winxed, NQP-rx, NQP and Rakudo. We’re a long way off but the progress is extremely rewarding.

Andrew Whitworth: Packfile Tags Cleanup

2012-03-04T08:00:00+00:00

I’ve complained a lot about Parrot’s packfiles and packfile-related subsystems. A few months ago I started writing about a plan to fix some of these issues. Now, after something of an extended hiatus from hardcore parrot hacking, I’ve decided to get started on doing all this work.

The biggest part of my plan, from a user-interface perspective was changing this:

.sub 'foo' :load
    ...
.end

to this:

.sub 'foo' :tag('load')
    ...
.end

Also, as part of this change, we want to change this:

load_bytecode "bar.pbc"

to this:

$P0 = load_bytecode "bar.pbc"
$P1 = $P0.'subs_by_tag'("load")
$I0 = $P1.'is_initialized'('load')
if $I0 goto done_initialization
$P2 = iter $P1
loop_top:
unless $P2 goto loop_bottom
$P3 = shift $P2
$P3()
goto loop_top
loop_bottom:
$P1.mark_initialized("load")
done_initialization:

I’ve discussed these changes before but I will repeat one important detail that is well worth repeating: Despite the fact that the total amount of PIR code increases, the overall performance profile improves. The second set of code snippets are faster to execute and are much more flexible. The performance improves as the number of :load or :init subs in your packfile increases. You gain the ability to tag subroutines with any string name you want, use as many different string names for subroutine groups as you want, get a list of subs to execute in any order at any time, be able to initialize and reinitialize libraries at any time as many times as you want, and do a few other cool and empowering things. Actually, the system is set up to allow any arbitrary PMC in the constants table to be tagged with a string name and accessed using that name, but the only thing we have syntax for right now are the Subs.

Another point that I also feel like repeating is this: People shouldn’t write PIR code by hand. Don’t do it. You will write something friendly like Rakudo Perl 6, or NQP or Winxed, and those compilers will automatically generate the necessary PIR sequences for you. It will take some effort on my part to get the various compilers updated, but the end user shouldn’t see any differences. So long as you aren’t writing your code in PIR directly, you’ll be fine.

So what’s involved in making this change? The packfile system, especially the packfile loader and the bits that are intertwined with IMCC are some of the messier or at least most obscure parts of Parrot. This is one of the subsystems of Parrot that is the least abstracted and requires the most low-level bit twidling. The GC is a comparable beast in this regard. Things like memory alignment, offsets, complicated nested struct definitions and accesses, byte ordering, and a variety of other similar concepts are very common in this subsystem. Any changes must be made with extreme care.

At the time of this writing I’ve already made several major changes in my branch and have several more to go. I’ve removed the :load and :init flag syntax from the IMCC lexer and parser (they’ve been deprecated for a long time now and a replacement syntax has been available for some months). I’ve removed the awful do_sub_pragmas and PackFile_fixup_subs functions and rearranged significant amounts of logic. I’ve also taken a shotgun approach to converting :load and :init flags in the various PIR code libraries to :tag("load") and :tag("init") respectively. Some of these conversions were automated and a little bit over zealous, but that is to be expected at this stage of the game. Code for handling :immediate and :postcomp flags have been moved into IMCC where they belong. libparrot now has no direct knowledge of either of those two things.

Coming up are several more big changes before this branch is even remotely close to being mergable:

I need to clean up interp initialization logic that had been spread around in various places. The embedding API defines a central execution path for running bytecode programs, and the initialization logic needs to be moved higher up along that path.
I need to set a reference to the owning PackfileView PMC in each loaded Sub PMC. Currently we anchor every single PackfileView PMC to prevent them from ever being collected by GC. By storing a reference to the Packfile in the Sub PMCs, we can mark reachable Subs, mark reachable Packfiles, and therefore allow old, unreferenced packfiles to be GC collected. This fixes an old and sizable memory leak (especially in programs that do a lot of dynamic compilation. If we do this in the right way in the right places, we can close the memory leak and reduce iteration over packfile constants looking for Subs and PMCs to mark and keep track of.
I need to rewrite Parrot_load_language and friends. This is the function that implements the load_language opcode behavior. A “language” in this context is a library package which usually contains a compiler object, a runtime, and any dynpmc and dynop libraries required by those. This codepath shared much logic with Parrot_load_bytecode (the guts behind the old load_bytecode_s opcode variant), and therefore shares many problems and misbehaviors.
I want to refactor the relationship between the PackFile* structure and the PackfileView PMC. If the PMC is the GCable wrapper around the structure, the whole thing is more easily managed by GC if we can guarantee that the PMC always exists if the struct does. Refactors here can further clean up code in the packfile loader and elsewhere.
We need to rip out a lot of old, dead, crufty code that is no longer needed. This includes things like Parrot_compile_file and Parrot_compile_string, which are unnecessary internally because we have easily accessible and easily usable compiler objects at the PIR level. This also includes various helper routines for Parrot_load_bytecode, Parrot_load_language and do_sub_pragmas, helper routines for working with :load and :init flags
I need to update Winxed, NQP-rx, NQP, Rakudo and any other HLLs and libraries that still rely on the old behaviors.

So there are plenty of steps left before we can talk merger, and I don’t expect to get this work anywhere near master before 4.2. It’s a lot of work but it’s fun and will be rewarding when it’s all done. I’m looking forward to getting this all wrapped up in the coming weeks and months.

parrot.org: Parrot 4.1.0 "Black-headed Parrot" Released

ayardley — 2012-02-22T00:40:53+00:00

The Black-headed Parrot is ... most often found in pairs or small noisy flocks of up to 10 individuals, but sometimes up to 30.
--http://en.wikipedia.org/wiki/Black-headed_Parrot

On behalf of the Parrot team, I'm proud to announce Parrot 4.1.0,
also known as "Black-headed Parrot". Parrot
is a virtual machine aimed at running all dynamic languages.

Andrew Whitworth: Introspection, Disasembly and PACT (GSOC)

2012-02-21T08:00:00+00:00

In my post a few days ago I mentioned Google Summer of Code 2012 and gave a lightning list of simple project ideas that might be worth pursuing. Today I’m going to expand on one of these ideas because it’s fertile ground for many possible GSOC projects, including the possibility of several projects concurrently if we have multiple students interested in it.

Parrot has a lot of introspection ability, but we don’t really have the tools necessary to introspect bytecode. We need some kind of tool that, given a Sub PMC or a PackfileView PMC or similar will be able to provide a disassembled representation of the actual opcodes. Here’s a basic code example of what I am talking about:

function foo() { var x = 2; ... }

function bar() {
    var disassem = new Parrot.PACT.Disassembler();
    using foo;
    var raw = disassem(foo);
    var reg = raw.registers();          // Get register counts
    var lex = raw.lexicals();           // Get info about lexicals
    var constants = raw.constants()     // Referenced constants
    var ops = raw.opcodes();            // Symbolic Opcodes
    say(ops[0]);                        // "set_p_i, $P0, 2", etc
}

These are just some random ideas and not all of them are necessary to implement. The most important part, in my mind, is getting a list of symbolic Opcode PMCs. Each Opcode PMC would have this general form:

class Parrot.PACT.Opcode {
    var opname;         // The name or short name of the op
    var opnumber;       // The number of the opcode
    var oplib;          // The oplib which owns it
    var args;           // Array of Arguments
                        // Either Register or Constant
    ...
}

I prefix the disassembler classname with the namespace “Parrot.PACT” because eventually this should be an integral component of the PACT library. When we use PACT to assemble packfiles (and, ultimately, bytecode files) we’ll be constructing a list of these Opcode PMCs and then using a serializer to write them down to raw bytecode.

                  Serializer
Array of Opcodes ------------> Packfile Bytecode Segment

          Deserializer
Bytecode --------------> Array of Opcodes

An excellent proof of concept system would combine these two mechanisms together into a faithful round-trim assembly/disassembly mechanism. In fact, there are multiple little potential projects here that can be arranged and ordered/prioritized to create a summer-long project or many:

Create a tool to disassemble raw bytecode into Opcode PMCs, and create a disassembler program to interact with the user and print the disassembly listings to file/console.
Create a tool for round-trip disassembly and assembly. Write the disassembly type, then write a tool that does the reverse operation (take a list of Opcodes and write a valid Packfile or bytecode segment).
Create the tool to disassemble raw bytecode, then write a utility layer to construct a control flow graph from those Opcodes. This layer could be used in turn to create things like code complexity analyzers, or even simple decompilers (for the very ambitious student).
Write a tool to take a stream of Opcode PMCs and other related data (tables of constant values, annotations, debugging symbols, etc) and write them into a valid and executable packfile. This would be the base layer of the PACT assembly engine, and would be used to help build compilers and other tools.
Construct anything else PACT-related (AST and manipulators, CFG/DAG and friends, PIR->Opcode assembler, etc). There is lots of fertile ground here for projects (and we have a lot of ideas and designs already put together for these things).

There are lots of ideas here and I’ve still only scratched the surface. My goal with this post is to show how fertile this ground is, how much available work there is to be had and how many new features we desperately need.

Here’s a basic flow graph of things I’m envisioning as eventual parts of PACT or its close cousins. This will show the kinds of components that PACT may either eventually contain or serve as the common substrate for:

                                      PIR and PASM Code
                                              |
            Optimizers         Analyzers <-+  |  +-> Debugger/Live
            ^ |    ^ |           ^         |  |  |            Interpreter
            | V    | V           |         |  V  |
HLL code -> AST -> Control Flow Graph -> Opcode stream -> Packfile
             ^       |           ^         |  ^  ^           |
             |       |           |         |  |  |           |
            HLL <----+           +---------+  |  +-----------+
       (Decompiled)                           |              |
                                              |              |
                                             PIR Code <------+
                                          (Disassembly)

One day when I have more time I may try to put this together into a real image of some variety. ASCII graphics were good enough for our digital ancestors and they will suffice here for a first draft. As you can see this graph contains several components, any one of which or any small subsection might make for an interesting and extremely rewarding project over the summer. This also ignores the inherent complexity and layered architecture possible in things like the AST transformations and optimizations, register allocation, etc. My point is that even the blocks on the graph above can be further decomposed into a variety of smaller but still interesting projects. If any of this stuff looks interesting to you, please get in touch ASAP so we can start talking and planning. Obviously this is more work than one person will do in one summer, so we want to make sure we are coordinating between all interested parties.

I think that if we start on the left side of this chart and implement the routines for reading from and writing to packfiles first, we can start building layers of additional functionality on top of them. This gives us an ability to break such a big system up into managable parts, to complete some of those parts in small summer-sized chunks, and to be able to use intermediate implementations to solve real problems while we wait for the rest of the system to grow and mature.

If we had multiple students interested in working on PACT in one capacity or another it would be an awesome way to maximize developer resources and help push forward the idea of code reusability. I’m really excited about this whole area and would love it if some students were interested in it too.

Andrew Whitworth: Compile Winxed With Winxed

2012-02-18T08:00:00+00:00

You can compile Winxed code with Winxed itself! What’s that you say? The Winxed compiler is bootstrapped and self-hosted, and is written in Winxed and already compiles winxed? Well, that’s all true. Sort of. However there is one small caveat: The Winxed driver program historically has not been able to perform the last step of compilation. The driver compiles winxed code down to PIR, but then uses the spawnw opcode to invoke an instance of Parrot to compile the PIR down to PBC.

I’m pleased to say that this last step is no longer necessary. At least, not in Winxed master (which has not yet been snapshotted into Parrot core).

Here’s a small toy compiler driver that uses Parrot’s PackfileView PMC to compile a .winxed file down into .pbc without spawning any child processes:

function get_winxed_compiler(string pbc_name = "winxedst3.pbc")
{
    var wx_pbc = load_packfile(pbc_name);
    for (var load_sub in wx_pbc.subs_by_tag("load"))
        load_sub();
    return compreg("winxed");
}

function main[main](var args)
{
    var wx_compreg = get_winxed_compiler();
    string winxedcc_name = args.shift();
    string infile_name = args.shift();
    string outfile_name = args.shift();

    string code = (new 'FileHandle').readall(infile_name);
    var pf = wx_compreg.compile(code);
    pf.write_to_file(outfile_name);
}

That’s less than 20 lines of Winxed code to get the Winxed compiler object loaded, to compile the code and to output the PBC to file. We can make this better, of course, by being more flexible in the handling of arguments and printing out basic help and error messages and all that stuff. Eventually we are going to update the winxed executable itself to use this trick instead of spawning the child process. This should, I hope, have a noticably beneficial effect on compiling with Winxed from the commandline. For large, long builds like Rosella has, any speed improvements are appreciated.

One particularly interesting tidbit to notice is the very first line: A new syntax for handling optional parameters. I put a patch for that feature together last week and NotFound decided he could do the same thing but better than I did. So, the latest Winxed compiler (again, not yet snapshotted into Parrot Core) supports this syntax for providing default values to optional arguments. I hope that this feature is included with the 4.1 release next week. Here are some examples of the new feature in action:

// This...
function foo(var bar [optional], int has_bar [opt_flag])
{
    if (!has_bar)
        bar = default_bar_value();
    ...
}

// ...is the same as this
function foo(var bar = default_bar_value())
{
    ...
}

The initializer can be any expression value, including expressions involving previous arguments:

function foo (var bar, var baz = bar.some_method(bar))

This new syntax probably has a few kinks to work out still, but it’s a very cool and very appreciated addition. I’m hoping to use this new syntax to clean up a lot of code in Rosella and hopefully in Jaesop too.

Andrew Whitworth: Compiling Templates

2012-02-16T08:00:00+00:00

I had a precious few hours to myself yesterday and was able to do some updating work on Rosella’s Template library. I was able to use the time to implement a feature I had wanted for a while: template compilation. You can now compile a template into winxed code or even compile it all the way down to a Packfile. Actually, that’s sort of a lie. The Winxed compiler .compile() method returns an Eval PMC not a PackfileView PMC. I’m going to submit a patch for that soon, and when I do you’ll be able to save the template to a .pbc file.

Here’s how to use the Template library in the basic, interpreted way:

var engine = new Rosella.Template.Engine();
string output = engine.generate(template, context);

The template is a string with a format that I’ve demonstrated before, and the context is any user-defined data structure that you want to use to populate the variables in the template. I won’t go into detail about those things in this post. Now, after my recent changes and additions, you can compile your template to an executable Sub:

var engine = new Rosella.Template.Engine();
var sub = engine.compile(template);
string output = sub(context);

Or, if you really want to see the generated winxed code, you can get that:

string wx_code = engine.compile_to_winxed(template);

The compilation process does take some time, there’s no way to deny that. There are ways to mitigate that expense, of course. You can compile ahead of time and save the code to a file or even a .pbc and execute that later. There are several strategies if you’re really interested, I won’t go into too much detail here. Once the code is compiled, which can and should be done ahead of time, the time savings during execution are significant. Here’s some benchmarking I’ve done to time a relatively simple template with a ten thousand iteration <$ repeat $> loop:

Interpreted:
0.969569s - %100.000000
0.796563s - %82.156419 (-%17.843581 compared to base)
0.900937s - %92.921402 (-%7.078598 compared to base)

Compiled:
0.365500s - %37.697161 (-%62.302839 compared to base)
0.498571s - %51.421914 (-%48.578086 compared to base)
0.367616s - %37.915399 (-%62.084601 compared to base)

In some cases, pre-compiling the template can take a third as much time as interpreting the template directly. Almost every timing I’ve seen is around 50% or better of the interpreted time.

This is a very new feature and I haven’t added it to the test suite yet. Expect rough edges as I play with it and optimize it. If you’re doing a lot of templating with Rosella, this feature could help save some time for you and plus it was a really fun thing to work on.

Andrew Whitworth: GSOC 2012 Is Starting

2012-02-15T08:00:00+00:00

You might not see it, but I’m starting to get very excited. Discussions about the Google Summer of Code program is starting up for the 2012 summer. Projects in years past have lead to some awesome developments in Parrot, either directly or indirectly, and 2012 could easily deliver more.

For prospective students, here are some blog posts I’ve written in years past about the process, and what you need to do to get started:

If you’re interested in participating in GSoC this year, with Parrot or any other organization, I suggest you read those two posts. They’re important. Many students in school or freshly graduated may know how to code but might not know much about some of the tangential topics: source control (git, in the case of Parrot), documentation, unit testing, refactoring, etc. Now would be a great time to start brushing up on all those topics, so you don’t waste time during the summer getting your tools in place.

As usual I will try to post some ideas for projects here on this blog. If you are an eligible student and you are interested in one or more of these ideas please get in touch with me or other interested Parrot developers. If you have other ideas that I don’t mention, that’s cool too. Get in touch anyway and we can start talking about those ideas. The important thing is to talk to us. Seriously, it’s important.

Here are a few ideas off the top of my head that might be worth some more investigation. I might write additional posts about these ideas if people want more information about them:

Opcode disassembler. We have a growing set of tools for working with bytecode from running Parrot code, but we don’t have all the pieces that we would need to make a fully self-hosted disassembler. What we still need are tools to read out the raw bytecode and convert into a sequence of Opcode PMC representations, then a driver program to turn that output into readable (and hopefully round-trim compilable) disassembly output.
PACT An alternate to Parrot’s venerable PCT library, called PACT, has been in the planning stages for many months. What we need is somebody with the time and motivation to start putting those ideas into practice. A toolkit library that works with syntax trees and outputs working bytecode libraries would be quite an awesome thing to have, and would help us push the state-of-the-art for Parrot compilers up a few notches. Intended to be a layered system, the successful student could implement only a few of the necessary layers.
Jaesop Stage 1 Jaesop is a bootstrapped JavaScript compiler. Right now there is a stage 0 compiler which uses node.js to compile JS into Winxed. We need a stage 1 compiler, written in JavaScript, that can run on stage 0 and compile itself. It’s an interesting, mind-bendy kind of project. If you know compilers and you like JavaScript, this might be the project for you.
Anything Python-Related Parrot wants and needs more Python love. We’ve had a few attempts at making a working Python compiler before on Parrot. Working on any of those, starting a new attempt, or working on other ways to integrate Python and Parrot would all be greeted with some eagerness.
New Object Model Parrot needs a new object model. Right now we have 6model waiting in the environs, but we haven’t integrated it yet. There is some question about whether we want to copy+paste merge 6model as it is, or if we want to try and make a more custom adaptation of it from the ground up. Proposals to do either, or anything closely related, would be very interesting.
LAPACK Bindings for PLA I’ve wanted PLA to get LAPACK bindings for a long time now, and I’ve never had the time to do it myself. I’ve never really even had the time to design what such a thing would look like. I suspect we can do the lion’s share through raw NCI. Tools to solve matrices for eigenvalues and eigenvectors and common transformations and decompositions would be very interesting indeed.
New PackFile Loader I’ve complained about the unnecessarily magic behavior of our packfile loader before. A new, re-written packfile loader would not automatically assemble Namespace or MultiSub PMCs at load time. Down this path lay the possibility for significant performance improvements and complexity reductions in some of our oldest and least-friendly code. It would require serious changes to IMCC, PIR, and higher-level compilers like Winxed. The new NQP already does the right kind of thing and would serve as a great example to follow.
Anything Thread or Asynchrony Related Parrot has Green Threads on Unixy platforms. Extending proper support to Windows and elsewhere would be awesome (again, something I want to do but have not had time or a Windows machine for testing). Asynchronous IO using new threading primitives and anything else that you could think to build with the new threading system would be awesome. Adding in types that would help the effort such as lock-free arrays and hashes would be nice assets. Adding in support for concurrency primitives like locks, mutexes and critical sections would also be cool (and implementing those in terms of green thread primitives would be even cooler).
Divided VTABLE Parrot’s VTABLE is a huge monolithic structure, and there have been many suggestions recently that we break it down smaller chunks based on roles. The “Number” role would contain arithmetic vtables, while the “invokable” role would have VTABLE_invoke and friends. GCable role, Array role, Hash role, Metaobjecet role, etc. These are all things that we could use and would decrease overall memory footprint and increase flexibility in the system. Bonus points if this work was integrated closely with 6model and other proposed changes in our PMC subsystem.

These are just a few of the ideas I have on the top of my head this morning. Some, I’m sure, are too big. Others are too small. But in each is the kernel of a good idea and if anybody reading this is interested we should start the conversation now to get these vague ideas focused into compelling proposals.

Andrew Whitworth: Rosella Reflect

2012-02-01T08:00:00+00:00

Earlier this month I released the new Reflect library in Rosella. I hadn’t mentioned it before, but the library is sufficiently interesting that I want to talk about it at least a little bit. The Reflect library adds in tools for reflection. Somewhere, an etymologist weeps a tear of joy for the creative naming, I’m sure.

The Reflect library adds in wrappers for classes and packfiles that makes them easier to work with for many operations. First, I’d like to use a couple code examples to show the most basic API:

// Get the Sub PMC that we're currently executing
var s = Rosella.Reflect.get_current_sub();

// Get the current context
var c = Rosella.Reflect.get_current_context();

// Get the current object, if the current Sub is a method call
var obj = Rosella.Reflect.get_current_object();

// Get the class of the current object, if the current Sub is a method call
var c = Rosella.Reflect.get_current_class();

// Get a Module object for the packfile where the current Sub is defined
var m = Rosella.Reflect.get_current_module();

// Get a reflection wrapper object for the given Parrot Class PMC
var r = Rosella.Reflect.get_class_reflector(myClass);

// Get a Module object for the packfile in "foo/bar.pbc", loading it as
// necessary
var m = Rosella.Reflect.Module.load("foo/bar.pbc");

That’s the basic API that the library provides to get basic information about where execution is happening at the moment when the call is made. Once you have a Module object or a Class reflector object, you can do all sorts of cool things that used to be a pain in the butt to do manually:

var m = Rosella.Reflect.get_current_module();
say(m);          // Stringified, produces the name and version of the packfile
m.load();        // Execute all :tag("load") and :load functions
n.init();        // Execute all :tag("init") and :init functions
say(m.version(); // Get the version string of the packfile "X.Y.Z"
say(m.path());   // The on-disk path to the current packfile

// Get a hash of all Class PMCs defined at compile-time (using the :method
// flag on Subs) defined in the packfile, keyed by name
var c = m.classes();

// Get a list of all non-:anon functions defined in the packfile
var f = m.functions();

// Get a hash of all non-:anon functions in the packfile, organized into
// a hash keyed by namespace
var f = m.functions_by_ns();

// Get a hash of all NameSpace PMCs defined at compile-time
var ns = m.namespaces();

Once you have Class and NameSpace PMCs from the packfile, you can start to do all sorts of cool operations and analyses on them. Once you have a Class reflector object, you can do even more stuff with that:

var c = Rosella.Reflect.get_current_class();

// Create a new object of the current type
var o = c.new();

// Say the name of the class
say(c.name());

// Attributes are encapsulated as objects. You can get an Attribute
// reflector and use it later to get and set values on objects of this
// type or subclasses
var attr = c.get_attr("foo");
var value = attr.get_value(o);
attr.set_value(o, "whatever");

// Methods are also encapsulated. You can get a method reflector now and
// invoke it on objects later (including objects of different types)
var method = c.get_method("bar");
var result = method.invoke(o);
var meths = c.get_all_methods();

// Basic capability detection. Determine if objects are members of the
// class or their subsets, and determine if the class can perform certain
// methods
if (c.isa(o)) { ... }
if (c.can("bar")) { ... }

I hope the code examples make up for the terse explanations.

The Reflect library is currently focused on reading data from things like Classes and Packfiles, not on creating these things like the new PACT project is supposed to do. I want to extend this library even further with abilities to further introspect functions down to the opcode level and then…Well, when we have a stream of opcodes to analyze the possibilities are endless. I’d also like the ability to get better introspection of the interpreter and global state, though a cleaner interface than the hodge-podge of interpinfo opcodes and ParrotInterpreter PMC methods and whatever else we currently use.

As always, using the interface Rosella provides will help to insulate you from changes to the various underlying mechanisms when we finally get around to cleaning them up and making them sane. There isn’t a huge push to make such cleanups on a large scale yet, but I wouldn’t be surprised if a few things started getting prettified in the coming months at a slow pace.

I’ve already started using the new library in several of the Rosella utility programs such as those that create a winxed header file or a test suite from an existing packfile. In all cases the updated programs are both cleaner and have more functionality than the previous incarnations. Expect to see this library improve and grow in 2012 and beyond, and expect to see it work closely with PACT, once that project gets moving forward.

Andrew Whitworth: Rosella Test Updates and Upgrades

2012-01-30T08:00:00+00:00

The first half of this month was dominated with some epic illnesses between my family members and myself, family functions and home maintenance. What little spare time I’ve had otherwise has been devoted to writing code, as opposed to writing blog posts about writing code. The blog has suffered.

The past couple days I’ve been working on Rosella’s Test library. It’s an old but good library and is, as far as I am aware, the most full-featured and easy to use testing tool in the Parrot ecosystem. With some of these most recent changes the library is better still.

Matchers

Kakapo had a series of Matcher routines and objects as part of it’s testing facilities, and for a long time I’ve been wanting to port some of those ideas over to Rosella. As of last week, I have a simple version of them. Matchers allow you to ask the question “is this thing like that thing”, with a custom set of rules. Let me give a basic example.

Previously in Rosella if you were unit testing a method which returned an array and you wanted to check that the array contained the right values, you would have to do something like this:

var result = obj.my_method();
var expected = [1, 2, 3, 4, 5];
self.assert.is_true(result != null);
self.assert.equal(elements(result), elements(expected));
for (int i = 0; i < elements(expected); i++) {
    self.assert.equal(result[i], expected[i]);
}

That’s a lot of work, although you can cut it down a little bit if you know for certain that the array isn’t null. With the new matcher functionality, you pass in two arrays and the Test library will match them for you:

var result = obj.my_method();
self.assert.is_match(result, [1, 2, 3, 4, 5]);

Internally the Test library maintains a list of matchers by name. When you pass in two objects, it loops over the list looking for a matcher that can handle the pair. In this case, one of the default matchers the library provides looks for objects which implement the "array" role, and then does element-wise matching on them. Another similar matcher does the same for hash-like objects that implement the "hash" role.

Another matcher checks to see if one or both of the two objects are strings, and then does a string comparison on them (converting the other, if it isn’t a string already) and the last of the default matchers is used to compare floating point values with a certain error tolerance.

Since matchers are stored in a hash, you can access them by name, delete them, add your own, and replace existing ones if you want new matching semantics. This is especially useful in something like Parrot-Linear-Algebra, where I can say

$!assert.is_match($matrix_a, $matrix_b);

…and the library will automatically compare the dimensions of the matrices and the contents of them without needing nested loops and other distractions.

Nested TAP

Another item I’ve had on my wishlist for a while now has been nested TAP. I’ve always wanted to support it, and in theory at least the system was designed modularly enough to generate it without too much hassle. Last weekend I put on the finishing touches and now am proud to say that Rosella.Test can run nested tests and generate nested TAP. At the moment the interface to use it is a little ugly (I’m actively soliciting feedback!), but the capabilities are all there:

function my_test_method()
{
    self.status.suite().subtest(class MySubtestClass);
}

function my_vector_test_method()
{
    self.status.suite().subtest_vector(
        function(var a, var b) { ... },
        [1, 2, 3, 4, 5]
    );
}

function my_list_test_method()
{
    self.status.suite().subtest_list(
        function(var test) { ... },
        function(var test) { ... },
        function(var test) { ... }
    );
}

Here’s an example of output from a similar test file in the Rosella suite:

1..4
    1..2
    ok 1 - test_1A
    ok 2 - test_1B
    # You passed all 2 subtests
ok 1 - test_1
    1..3
    ok 1 - test 1
    ok 2 - test 2
    ok 3 - test 3
    # You passed all 3 subtests
ok 2 - test_2
    1..5
    ok 1 - test 1
    ok 2 - test 2
    ok 3 - test 3
    ok 4 - test 4
    ok 5 - test 5
    # You passed all 5 subtests
ok 3 - test_3
    1..1
    not ok 1 - test 1
    # failure
    # Called from 'fail' (rosella/test.winxed : 481)
    # Called from '' (t/winxed_test/Nested.t : 40)
    # Called from '' (rosella/test.winxed : 1589)
    # Looks like you failed 1 of 1 subtests run
ok 4 - test_4
# You passed all 4 tests

The fourth test expects a failure in the subtest, which is why it says it passes when there is clearly some failure diagnostics appearing. This brings me to my next point…

Cleaner Diagnostics

Before when you ran a test and had a failure, you might see something like this:

not ok 2 - ooopsie_doopsies
# objects not equal '0' != '1'
# Called from 'throw' (rosella/test.winxed : 851)
# Called from 'internal_fail' (rosella/test.winxed : 1853)
# Called from 'fail' (rosella/test.winxed : 481)
# Called from 'equal' (rosella/test.winxed : 577)
# Called from 'ooopsie_doopsies' (t/core/Error.t : 18)
# Called from 'execute_test' (rosella/test.winxed : 1455)
# Called from '__run_test' (rosella/test.winxed : 1483)
# Called from 'run' (rosella/test.winxed : 1392)
# Called from 'test' (rosella/test.winxed : 1747)
# Called from '_block1000' (t/core/Error.t : 7)
# Called from '_block1177' ( : 158)
# Called from 'eval' ( : 151)
# Called from 'evalfiles' ( : 0)
# Called from 'command_line' ( : 0)
# Called from 'main' ( : 1)
# Called from '(entry)' ( : 0)

That’s a huge mess, and it’s a mess from two sides. At the top of the backtrace, you see all sorts of Rosella internal functions involved in the assertion and error handling. The bottom half of the backtrace is devoted to entry-way stuff. In this case there’s NQP-related entry code and then the Rosella entry code. You, as the test writer, don’t care about any of that. All you care about is the code you wrote and where its broken. If you have to dig through a huge backtrace to figure out where the error is, that’s a big waste of time and effort.

Now, Rosella filters that crap out for you. Here’s the same exact failure with the new backtrace reporting:

not ok 2 - ooopsie_doopsies
# objects not equal '0' != '1'
# Called from 'equal' (rosella/test.winxed : 577)
# Called from 'ooopsie_doopsies' (t/core/Error.t : 18)

Here you see the important parts of the backtrace only: The parts you wrote and the one assertion that failed. You don’t see the internal garbage, you don’t see the entry-way garbage, because those things aren’t of interest to the test writer.

Parrot-Linear-Algebra

Another small project I did a few days ago was getting the PLA test suite working again. It’s a testament to how stable both BLAS and Parrot’s extending interfaces are. Recent Rosella refactors removed some of the special-purpose features that existed only for PLA and for no other reason (and which were a pain in the butt to maintain). I fixed up the test suite and PLA builds and runs perfectly now.

That’s what I’ve been up to this month. I’m mostly done with my cleanups to the Test library now, barring a few more interface improvements I want to make. After that I’ve got a few projects to tackle inside libparrot itself. I’ll write more about those topics when I have something to say.

parrot.org: Parrot 4.0.0 "Hyperstasis" Released!

Whiteknight — 2012-01-18T02:40:41+00:00

At one extreme, it is possible to approach the subject on a high mathematical
epsilon-delta level, which generally results in many undergraduate students not
knowing what's going on. At the other extreme, it is possible to wave away all
the subtleties until neither the student nor the teacher knows what's going on.

-Stanley J. Farlow, Preface to Partial Differential Equations for Scientists and
Engineers

On behalf of the Parrot team, I'm proud to announce Parrot 4.0.0,
also known as "Hyperstasis". Parrot

Andrew Whitworth: Parrot 4.0.0 "Hyperstasis" Released!

2012-01-17T08:00:00+00:00

At one extreme, it is possible to approach the subject on a high mathematical epsilon-delta level, which generally results in many undergraduate students not knowing what’s going on. At the other extreme, it is possible to wave away all the subtleties until neither the student nor the teacher knows what’s going on.

-Stanley J. Farlow, Preface to Partial Differential Equations for Scientists and Engineers

On behalf of the Parrot team, I’m proud to announce Parrot 4.0.0, also known as “Hyperstasis”. Parrot is a virtual machine aimed at running all dynamic languages.

Parrot 4.0.0 is available on Parrot’s FTP site, or by following the download instructions at http://parrot.org/download. For those who would like to develop on Parrot, or help develop Parrot itself, we recommend using Git to retrieve the source code to get the latest and best Parrot code.

Parrot 4.0.0 News:
    - Core
        + Several cleanups to the interp subsystem API
        + Cleanups and documentation additions for green threads and timers
        + Iterator PMC and family now implement the "iterator" role
        + A bug in Parrot_ext_try was fixed where it was not popping a context correctly
    - Documentation
        + Docs for all versions of Parrot ever released are now available
          at http://parrot.github.com
    - Tests
        + Timer PMC tests were converted from PASM to PIR

The SHA256 message digests for the downloadable tarballs are:

a1e0bc3de509b247b2cea4863cc202cdceeaa329729416115d3c20a162a0dd88 parrot-4.0.0.tar.bz2
a63d45f50f7dd8ba76395cd2af14108412398ac24b8d827db369221cdb37fada parrot-4.0.0.tar.gz

Many thanks to all our contributors for making this possible, and our sponsors for supporting this project. Our next scheduled release is 21 February 2012.

Enjoy!

Andrew Whitworth: Rosella Date

2012-01-15T08:00:00+00:00

The Advent calendar idea ended up terribly. Let us never speak of it again. The holiday season was particularly busy this year with a higher-than-average load of family- friend- and work-related activities. Combine that with an unexpected (and absolutely unappreciated) invasion of a particularly unpleasant and long-lasting stomach bug, and you have something of a perfect storm. I won’t go into the details any further on this particular blog, but I will mention in passing that I’ve become extremely suspicious about the basic hygene habits of the other little turdbags that my son goes to daycare with.

To start 2012 off, and to get back into the swing of coding-for-pleasure, yesterday I went through and got the new Rosella Date library ready for prime time. The library is imperfect and incomplete, but those are things that can be fixed using the patented Andrew Style (tm) of meandering iterative development. For now, however, the library does seem to work well enough for basic tasks and it’s already proven to be a valuable tool for some tasks. Here I’m going to introduce the new library and talk about some of the things I changed in Rosella to support it and some of the ways I’ve already integrated it into the rest of the collection.

String Formatters

When you use sprintf to print out an integer (for instance) you can use some basic modifiers to control how the value is printed. %d prints out the basic version, but you can specify field width, padding, alignment and a few other details by using a format specifier like %-02d. Or, if you want to print the same value out as hex instead of base-10, you can use %x or %X, and use modifiers with those as well.

The problem with something more complex like a date/time representation is that sprintf is not able to handle them natively and some kind of mapping is needed.

Rosella has added a new StringFormatter type to the Core library to help with this problem. A StringFormatter is a type that takes an object and a format string and outputs a new string according to the two. The default StringFormatter uses sprintf internally, but other formatters may use different mechanisms and syntaxes.

var sf = new Rosella.StringFormatter();
string s = sf.format(my_obj, "This is %s");

As an aside, this does demonstrate the fact that our get_string vtable really is insufficient for a lot of purposes. I suggest that get_string should take a parameter for a format string. We could easily incorporate that into the default sprintf implementation like this:

$S0 = sprintf "%{foobar}p", $P0

In that invocation, the get_string vtable on the first parameter would be called with the string argument “foobar”. A normal invocation like this:

$S0 = sprintf "%p", $P0

…would call get_string with a null format and the behavior could be whatever the default string representation for that type is. By overriding get_string in your types to take different formats or to respond to common formats differently, you could have pretty detailed control over stringification at all levels.

Date Library

The new Date library provides a Date type for working with dates and times. You can create one in three ways:

var d = new Rosella.Date(t);
var d = new Rosella.Date(year, month, day);
var d = new Rosella.Date(year, month, day, hour, minute, second);

The first uses a system-specific time integer value to represent a time since the system epoch. This is the kind of value you get from the time opcode, or from stat calls on filesystem objects, for instance. In the third option, hours are specified on a 24-hour clock.

There are also a few functions you can call to get particular dates:

var d = Rosella.Date.now();
var n = Rosella.Date.min();
var x = Rosella.Date.max();

The first value should be the current date/time (as gotten from Parrot’s time opcode). The second value is a minimum date object which corresponds to the minimum possible date value to display and is guaranteed to be evaluated as less than any other date. The third one similarly is a maximum date value which corresponds to the maximum possible date and is guaranteed to always be compared greater than any other date.

Dates are immutable. Once you create them, they cannot be modified in-place. Instead, several operations are provided to perform operations and return new Date objects with the results. Here are some examples:

var d = Rosella.Date.now();
var e = d.add_seconds(20);
e = d.add_hours(15);
e = d.add_months(24);
e = d.add_years(1000);

In each case, the variable e becomes a new date value and d is left unmodified. Two other methods let you pick out just the date components or just the time components:

var n = Rosella.Date.now();
var d = n.date();
var t = n.time();

Finally, you can use a new DateFormatter type to format the date value into a proper stringification:

var d = Rosella.Date.now();
string s = d.format_string("yyyy - MM - dd and the time is: hh:mm:ss");

At the moment the formatter is dirt simple and only supports a few formatting codes such as yyyy, MM, dd, hh, mm, and ss. I will be making it much more useful in the coming days, if I can settle on an algorithm which doesn’t completely stink.

FileSystem

The FileSystem library now returns Date objects from certain file-time methods:

var f = new Rosella.FileSystem.File("t/harness");
var ct = f.change_time();
var at = f.access_time();
var mt = f.modify_time();

In each case, the value returned from the stat call on the file is used to create a new Date object.

Query

Dates are completely comparible, so you can sort them and work with them like other values in an iterable:

var a = [
    Rosella.Date.now(),
    Rosella.Date.max(),
    Rosella.Date.min()
];
Rosella.Query.iterable(a)
    .sort()
    .foreach(function(d) { say(d); });

This toy example sorts the three Date values, with the min first, then now, then max, and prints them out to the console. The stringified versions of the special min/max dates aren’t particularly instructive, but they do get the point across.

So that’s the new Date library. It does need more functionality and definitely needs more tests, but it is working pretty well for me now and has already proven itself useful for a number of purposes. Expect to see more of it in 2012.

parrot.org: Parrot Documentation Revision Effort (Blog trois)

ayardley — 2012-01-11T05:07:21+00:00

This blog is to announce the completion of placing all of Parrot's documentation on 'parrot.github.com'. The documentation ranges from the present version (i.e., v3.11.0) to Parrot's release v0.0.6(0)[1]. To view the documentation, please navigate your preferred browser to http://parrot.github.com and select (or click) the "Parrot Documentation Releases (3.10.0 - 0.1.1)" link.

Thank you.

----------

Tadeusz Sośnierz (tadzik): State of Dancer on Perl 6

ttjjss — 2012-01-06T00:24:34+00:00

Bailador is growing better and bigger and starts to resemble a real tool more and more. Let’s see what new features it has gained recently.

Remember the first example in perldoc Dancer? It’s not really different in Bailador:

use Bailador;
get '/hello/:name' => sub ($name) {
    return "Why, hello there $name"
}
baile;

Aside of being a little Spanished, what did it give us? We have subroutine signatures in Perl 6 so we can pass the :name parameter to the sub; there’s no need to use param() now: it’s gone.

You don’t need to pass everything using GET of course. post keyword is also supported.

post '/' => sub {
    return request.params.perl
}

The above will print something like ("foo" => "bar").hash, if fed with appropriate request.

any() is a reserved keyword in Perl 6, and while you can use it, it means a completely different thing. Instead of any('get', 'post') you can just do it like this:

get post '/' => sub {
    if request.is_get {
        return "I am GET"
    } else {
        return request.params.perl
    }
}

post, as well as get return their arguments, so you can chain them like in the example above. It also shows the joy of request object, which you can use to inspect the request being processed. It’s not as cool as Dancer::Request, but it does the job, being quite small and simple.

What else do we have? Let’s show off a bit and write a simple-simple pastebin webapp.

use Bailador;

unless 'data'.IO ~~ :d {
    mkdir 'data'
}

get '/' => sub {
    template 'index.tt'
}

post '/new_paste' => sub {
    my $t  = time;
    my $c = request.params<content>;
    unless $c {
        return "No empty pastes please";
    }
    my $fh = open "data/$t", :w;
    $fh.print: $c;
    $fh.close;
    return "New paste available at paste/$t";
}

get /paste\/(.+)/ => sub ($tag) {
    content_type 'text/plain';
    if "data/$tag".IO.f {
        return slurp "data/$tag"
    }
    status 404;
    return "Paste does not exist";
}

baile;

Holy cow, what’s that! Let’s go there piece by piece. First, we’ll create a data directory if it doesn’t already exist. No black magic here, let’s proceed. What’s next? Templates! Here we just load index.tt, not passing any parameters, but that works too and some example apps use that in their example templates.

The handler of new_paste uses our well-known request object again, and creates a new file for a paste, identified by the current time.

The last get block uses some nifty features, so let’s take a look. It uses regexes, and you can see that they also cooperate with subroutine parameters without black magic. We then set a content_type as we’ll do in Dancer, and send status 404 if no paste have been found. Easy peasy? I suppose so. That’s it, it works like a charm.

Thus we’ve covered all the features in Bailador as for now. I don’t think it’s that poor, as for about 100 lines of code.

What’s next? What’s missing? You tell me. Or you contribute; the code is dead simple and implementing stuff like before(), after(), before_template() etc should be a matter of 3-5 lines, I think. Feel encouraged to look into the code and hack on it. If you have any questions, suggestions or criticism, don’t hesitate to tell, or poke me on #perl @ Freenode. Have fun!

Andrew Whitworth: Advent 9 - Jaesop

2011-12-29T08:00:00+00:00

Over the summer of 2011 we had a GSOC project that was trying to build a JavaScript compiler for Parrot. That project made some interesting progress but ultimately I think the approach ended up being flawed. That project attempted to convert JavaScript into PIR code, which is a pretty big gap in terms of both syntax and semantics.

Towards the end of the summer, after it was too late to go back and start from the beginning, I had a different idea: What if we tried to generate Winxed code instead of PIR code? Winxed would handle things like register allocation, and the Winxed syntax is so similar in some places that the translation could almost be verbatim copying. I put those ideas together with node.js and the cafe compiler and Jaesop was born.

Jaesop intends to be a full JavaScript on Parrot compiler. The plan is to write as much of it in JavaScript itself as possible, and bootstrap upwards. The first component, “stage 0” uses node.js to run a converter that compiles Javascript code into winxed. That’s the only part we have working right now, but what we do have is working pretty well.

In 2012 I want to get the next component, “stage 1” working. Stage 1 will use the stage 0 compiler to compile itself. It will be written in JavaScript (translated to Winxed en passant) and will be running entirely on Parrot. My goal for stage 1 is to be generating bytecode directly instead of generating either Winxed or PIR as an intermediary. That’s going to require some serious help from a compiler toolkit of some variety and I have very high hopes for using PACT for this purpose when it’s ready.

Stage 0 works very well and passes a small but interesting test suite I’ve set up for it. It does not self-compile, but there are only a few relatively small things standing in the way. For instance, regular expression support and pcre bindings are not complete yet, and the grammar currently requires semicolons at the end of statements but the code generated from Jison does not always contain semicolons. I also haven’t built in the require() function, which is used by the stage 0 compiler to load in the various code files. These are all small issues and with a small amount of work I expect stage 0 to be able to self-host. Whether I want to expend that effort or focus attention on stage 1 instead is a different question entirely.

In 2012, once PACT has matured and 6model has been integrated into Parrot I expect to get back to work aggressively on Jaesop. I’m looking for helpers too, in case anybody reading this wants to get involved in the development of a new JavaScript compiler. I’ll put out more of a call to arms when the prerequisites are in place and we’re ready to get the wheels turning on stage 1. I estimate that by spring or early summer we should be ready to get started on the next phase of Jaesop development.

Andrew Whitworth: Advent 8 - Rosella

2011-12-28T08:00:00+00:00

I had already written a post about Rosella for my woefully inadequate advent calendar, but it got lost to the sands of time (I think it’s on my other computer, committed but not pushed). Considering that I should have been on schedule to be on post #21 or #22 by now but am instead only on #8, I can’t really afford to not write a post when I have the ideas inside me. This post is also going to be about Rosella, and if I ever find the first one I may post that as well. I clearly haven’t had enough spare writing time this month to be even remotely picky about what I post or when.

I’m thinking I might like to try this little experiment again later in the year, when I’ve had more time to prepare and have fewer other things in real life demanding my undivided attention. Maybe we’ll shoot for some kind of christmas-in-July thing. Until then, I’ll happily conceed the “best Advent Calendar” crown back to moritz and the Perl peoples.

Rosella is a library project I started as a way to let me work on lots of different project ideas I had without having to duplicate build and test infrastructure. The goals from the project were solidified pretty early in the project lifecycle and have remained unchanged ever since: To provide solutions to common developer problems, in pure Parrot, in ways that are portable, reusable, and configurable. I’ve already talked about three of the oldest and most important libraries in the set: Test, Harness and MockObject on Day 5 of this Advent calendar. My other written-but-not-published post talked about three additional libraries as well: Query, FileSystem and Proxy. To avoid much duplicate effort in case I do ever find that post again, I’ll write about a few different libraries: Container, Random and Template.

The Container library is one of the oldest libraries in Rosella. It is an implementation of a dependency injection container for Parrot, and originally lived in it’s own Parrot-Container repository on github. My other library, for unit testing was also a separate repo. One day I was thinking about how I might like to use the Parrot-Container project to manage dependency injection in the Harness library, and trying to figure out how I wanted to manage the software dependencies. I wanted to use the Container library in the implementation of the unit test Harness, but I wanted to use the Harness and Test libraries to implement the test suite for the Container library repository. Combine that with a bunch of other ideas for new libraries I had brewing in the back of my mind and Rosella was quickly born.

The Container library is not quite as easy to use as a counter-part in the .NET or JVM environments because things are not always strongly typed, especially not at compile time or early in runtime when the container is initializing but the various bits of initialization logic in the program have not yet run. The Rakudo folks with their fancy object system and notion of gradual typing might have a different idea, but in terms of pure-parrot code, as Parrot exists right now, we can’t query a Sub PMC and ask it what types of parameters it expects to receive. For people who might be familiar with other dependency injection (or “inversion of control”) systems like Unity or Ninject, the syntax for setting up Rosella’s container type might seem a little verbose. It’s plenty powerful (and I have plenty of plans to upgrade things as Parrot’s threading support and object system improve in 2012 and beyond), but it is verbose:

var container = new Rosella.Container();
container
    .register(class Foo)
    .register(class Bar,
        new Rosella.Container.Resolver.TypeConstructor(class Bar, "Bar", 1, 2, 3)
        new Rosella.Container.LifetimeManager.Permanent()
    )
    .register(class Baz,
        new Rosella.Container.Resolver.TypeConstructor(class Baz, "BUILD"
            new Rosella.Container.Argument.Resolve(class Foo),
            new Rosella.Container.Argument.Resolve(class Bar)
        )
        new Rosella.Container.Option.Attribute("my_attr", "value"),
        new Rosella.Container.LifetimeManager.Thread()
    )
    .alias(class Baz, "Baz");
var b = container.resolve(class Bar);

The part that is significantly more verbose here than you would expect in Unity for example is the part where I have to specify the types of arguments to pass to the constructor. In a platform like .NET, the container can read the type metadata from the constructor object itself and make that decision. In Parrot, since we currently don’t make that kind of information available, you must specify. In Rosella’s offering you do have many of the features that you would expect from one of the more well-known containers: the ability to specify registration lifetimes, the ability to initialize objects by making method calls and setting attribute values after construction, the ability to specify global singleton instances, etc. It is a pretty cool tool, but I haven’t yet made as much use of it as I would have liked. In 2012 I expect to start integrating the container library into some of the other libraries, such as Harness, CommandLine and Template, to make user configuration easier and more straight-forward.

The Random library is a relative newcomer to Rosella, but is already demonstrating its usefulness in a variety of ways. The Random library was born from a few ideas I had turned into GCI tasks. An intrepid young student wrote an implementation of the Mersenne Twister algorithm for me in Winxed, and I set about writing up several other components such as a Box-Muller transform, a UUID generator, a Fisher-Yates array shuffler and a few other things. Now, you can do cool things with random numbers:

var prng = Rosella.Random.default_uniform_random()
int r = prng.get_int();

var uuid = Rosella.Random.UUID.new_uuid();
string id = string(uuid);

var a = [1, 2, 3, 4, 5, 6, 7, 8];
Rosella.Random.shuffle_array(a);

Implementations of things aren’t all perfect and there are a handful of bugs to be worked out (especially bugs resulting from arithmetic differences between 32-bit and 64-bit machines) but it is already very usable and reliable for the most part. If you need a random number generator, or a UUID generator, or other random-related things, this is a very nice tool to have available.

The Template library is something I wanted to write for a long time, but had to wait until all the prerequisites were in place first. Template is a text-templating engine library. You create an Engine object and feed it two basic pieces of information: The string template to use and the data context object to fill in the blanks. Presto-chango, out comes the complete text.

The Template engine can execute basic logical operations depending on the values in the data context, it can compile and execute inline snippets of code, it can load and assemble pieces of template from separate files, and do several other things that you would expect a templating library to do. I could write many examples of templates and their use, but I’ll stick with only one small example here for brevity:

Template:

Let's learn about <%
    var bar = context["animals"];
    return elements(bar);
%> new animals!
<$ for sound in animals $>
    The <# __KEY__ #> says <# sound #>
<$ endfor $>

Code:

string template = ...;
var context = {
    "animals" : {
        "Cow" : "Moo",
        "Bear" : "Growl!",
        "Cat" : "I CAN HAZ CHEESEBURGER"
    }
};
var engine = new Rosella.Template.Engine();
string output = engine.generate(template, context);

And the end result would be something like this (minus hash ordering concerns):

Let's learn about 3 new animals!
The Cow says Moo
The Bear says Growl!
The Cat says I CAN HAZ CHEESEBURGER

Rosella eats plenty of it’s own dogfood here. Several test files in the Rosella test suite and a few boiler-plate source code files are generated using the Rosella Template library. I’m planning to start using it to generate skeleton files for Rosella documentation as well. In the future, I may also use it to help with generating content for this very blog!

In 2012 Rosella is going to be adding a bunch of cool new stuff. Considering how far Rosella has come by now and the fact that it’s less than a year old, it’s kind of hard to speculate where we will be next year around this time. I’m planning to add a new Date/Time library within the next few weeks. I’m also planning a new reflection/packfile library, a benchmarking library, a code assertions library, and rewrites to several of the existing libraries to add new functionality and optimize performance in some key ways. Those are only my plans for the first two or three months of 2012!

Andrew Whitworth: Advent 7 - Google GCI and GSoC

2011-12-22T08:00:00+00:00

When you want to find something on the internet, Google is a pretty popular tool to use. When you want to get some code written, Google turns out to also be a pretty good idea.

Google has been doing it’s Summer Of Code (GSOC) program for several years now and every summer it’s a smashing success. Last year they started up with a new program called the Google Code In (GCI). That program, while wildly different from GSOC, is also incredibly awesome. Every year Parrot receives a huge amount of code from these programs, and we would not be where we are today without them.

The Summer of Code program is very straight forward: Every year we receive applications from college age and some highschool-aged students for projects. Over the course of the summer the accepted applicants work diligently and at the end, if they are successful, they get paid. Also there are cool T-shirts. The GCI program is aimed at younger students, mostly those in high school. Instead of one large project spanning several months, the GCI students work on a large number of bite-sized tasks from many organizations. Each task is worth points, and the students who have the most points at the end of the program receive a prize. Also, I think there are monetary rewards for completing a certain number of tasks.

Last year we had a huge number of tasks completed by several GCI students. We had many bits of code written or re-written, some new documentation written, and many many tests added. Our project-wide test coverage metrics increased dramatically, since we had several tasks devoted to test coverage and several talented young coders who were chasing them down. Once you figure out how to write tests, subsequent tasks go much more quickly. We had some students who, after getting the hang of things, were able to complete several tasks per day at the end.

I heard one or two accusations that Parrot was inflating point values by offering tasks that could be completed so quickly. I pointed out that many tasks were considered very hard by students at the beginning, but as their familiarity of the code increased the relative difficulty decreased. It wasn’t a problem of Parrot’s tasks being too easy, but the students learning and improving much faster than we could keep up with. By the end of the program we were learning that the difficulty really needed to ramp up over the course of GCI, because students would be capable of much more towards the end than they would be at the beginning.

This year things are going a little bit more slowly. We have fewer of the “increase test coverage” tasks, because our test coverage is still so high from last year in the core VM. I’ve scoured several potential sources of tasks including searching for TODO notes in the Parrot source code, scanning through our myriad of trac tickets, digging into ecosystem projects (Plumage and Rosella especially) and begging other contributors for task ideas. Already we’ve seen some very useful bits of code contributed to the project, including new features and impressive cleanups and refactorings of old crufty code.

GCI this year is essentially closed off. There were two opportunities to publish new tasks and those have both passed. Students are slowly working their way through the remaining tasks in the queue until the program ends in a few weeks. However, next year I’m sure we’re going to see both another GSOC and another round of GCI (At least, I hope we see them, since these are both so awesome!).

Despite being months away we’re already starting to look for project ideas and potential applicants for GSOC. Getting good ideas picked early allows us time to refine them, and getting familiar with potential applications now helps us to learn more about them and be more comfortable with them when time comes to make final selections. GCI doesn’t require so much preparatory work, but it would be nice if we went into next year with a larger pile of varied tasks for students to work on, instead of needing to scramble to create tasks in sufficient numbers at the last minute.

GSOC and GCI have both been amazingly successful programs in the past and I am hoping that trend continues into the future. Parrot has benefited from both programs to an amazing degree, and with a little bit of luck and a lot of planning we can keep the train moving into 2012 and beyond.

If you’re a highschool or college-aged coder, or know somebody who is, and would like to talk about getting involved with either program in 2012 (especially if you would like to work with Parrot specifically!) please let me know and I can make sure you get pointed in the right direction.

parrot.org: Parrot 3.11.0 "Duct Tape" Released

cotto — 2011-12-20T20:07:07+00:00

"If I had some duct tape, I could fix that."
 - MacGyver

On behalf of the Parrot team, I'm proud to announce Parrot 3.11.0, also known as "Duct tape". Parrot is a virtual machine aimed at running all dynamic languages.

Andrew Whitworth: Advent 6 - Embedding

2011-12-17T08:00:00+00:00

In late 2010 and early 2011 we spent a good amount of effort building a new embedding API for Parrot. I would like to say that the new API replaced an older, inferior API but that’s not really the case. We didn’t really have an old embedding API per se. We had a mismash of functions in a file called embed.c, but they hardly represented a consistent API, much less a complete set of things that an embedder would need. If anything the old embedding API was the entirety of all publicly exported functions from libparrot combined with a handful of utility functions that embedders in the past also needed.

In short, it was a mess. By early 2011 we had a much nicer API around to play with. Now that 2011 is almost over, the new API is considered to be extremely stable and robust.

Last December I started a project called ParrotSharp which embeds a Parrot Interpreter into the .NET CLI with C#. I haven’t shown that project too much love in recent months, but as of today it’s still building and seems to run correctly (although my IDE is telling me it can’t find NUnit on my system, so it won’t run my tests). That has to tell you something, when code I wrote months ago with the embedding API still works correctly even though so many things have changed in Parrot since then.

Parrot’s embedding API is a little bit verbose but very easy and straight forward to use. Also, all API functions return a true/false status value, so calls can easily be chained together. Here is an example of the embedding API in action:

int main(int argc, char** argv) {
    Parrot_PMC interp, bytecodepmc, args;
    Parrot_Init_Args *initargs;
    Parrot_String filename;

    GET_INIT_STRUCT(initargs);

    if (!(
        Parrot_api_make_interpreter(NULL, 0, initargs, &interp) &&
        Parrot_api_set_executable_name(interp, argv[0]) &&
        Parrot_api_pmc_wrap_string_array(interp, argc, argv, &args) &&
        Parrot_api_string_import(interp, "foo.pbc", &filename) &&
        Parrot_api_load_bytecode_file(interp, filename, &bytecodepmc) &&
        Parrot_api_run_bytecode(interp, bytecodepmc, args)
    )) {
        Parrot_String errmsg, backtrace;
        Parrot_Int exit_code, is_error;
        Parrot_PMC exception;

        Parrot_api_get_result(interp, &is_error, &exception, &exit_code, &errmsg);
        if (is_error) {
            Parrot_api_get_exception_backtrace(interp, exception, &backtrace);
            // Print out exception information to the console, or whatever
        }
    }

    Parrot_api_destroy_interpreter(interp);
    exit(exit_code);
 }

This is, essentially, a simple program to execute a bytecode file. However, it does show some of the basics of the embedding API. Every function returns a true/false, pass/fail status bit. All data types passed around are properly wrapped Parrot_PMC or Parrot_String types and it almost never uses any other raw pointer types. Also since we’re using PMC and STRING types, Parrot’s GC manages all the memory for you and you don’t need to be freeing things or cleaning things up (except for the interpreter itself).

This example above only shows a handful of API functions, but there are several dozen of them in the API and more can be easily added. We have API routines for performing a variety of actions on Strings and PMCs. We have API routines for loading, executing and writing bytecode. The API has decent defaults so you can just get an interpreter up and running quickly if you want, but it also have a variety of routines for tweaking and configuring the interpreter too. And, like I said (and I’ll say it a million times more if I have to) we can always add new methods to the embedding API if there is a need.

We have at least the basics of a C# wrapper projects in the wings, and I’ve been planning a proper C++ wrapper for a while too but I haven’t gotten around to it yet. That would make an excellent, smallish project for an intrepid newcomer to work on, especially one that knows C++. I like to think it should be easy to embed Parrot as a plugin for things like text editors or other pluggable unixy programs, but I haven’t taken the time to really dig into any of them yet. This might make another great project for an eager new parrot hacker.

Planet Parrot

parrot.org: Parrot 5.3.0 "W00tstock Parrot" Released!

parrot.org: The Parrot Foundation is accepted for GSOC 2013

Tadeusz Sośnierz (tadzik): Polish Perl Workshop status update

parrot.org: Parrot 5.2.0 "Stuffed Parrot" Released!

parrot.org: Parrot 5.1.0 "Zombie Parrot" Released!

cotto: It's Been Quiet

parrot.org: Parrot 5.0.0 "Johnny Five Alive" Released!

Tadeusz Sośnierz (tadzik): Threads for Rakudo Perl 6

parrot.org: Parrot threads on the perl6 advent calendar - Day 11

parrot.org: Parrot 4.11.0 "All together - Happy Birthday Lovebird" Released!

parrot.org: Parrot string encodings on the perl6 advent calendar - Day 7

parrot.org: Parrot 4.10.0 "Red-eared Parakeet" Released!

Andrew Whitworth: More IO Work?

parrot.org: Parrot 4.9.0 "Proto-Hydra" Released!

parrot.org: Parrot 4.8.0 "Spix's Macaw" Released!

Andrew Whitworth: September Status

Personal Status

io_cleanup1 Status

Threads Status

Other Status

Patrick R. Michaud: A Rakudo Performance

Andrew Whitworth: io_cleanup1 Lands!

parrot.org: Parrot 4.7.0 "Hispaniolan" Released!

Andrew Whitworth: Parrot 4.7.0 "Hispaniolan" Released!

parrot.org: Results

parrot.org: A new hope

parrot.org: Interpreters with butterfly wings

parrot.org: PACT: Adjusting the schedule

Andrew Whitworth: io_cleanup1 Done?

parrot.org: 4.6.0 Parrot "Wild Parrots of Telegraph Hill" Released

parrot.org: As long as hope remains

parrot.org: PACT: Spinning of the Wheels

Andrew Whitworth: IO Cleanups Home Stretch

parrot.org: Security API Update.

parrot.org: Pandora's Box

parrot.org: Progress till now

Andrew Whitworth: HTML with Rosella Xml and Net

parrot.org: PACT: Time flies when you're writing code.

Andrew Whitworth: Sockets and Encodings

parrot.org: Progress report

parrot.org: What have we done for you lately

parrot.org: 4.5.0 Parrot "Buff-faced Pygmy Parrot" Released!

parrot.org: Testing Part

parrot.org: Small Progress in Security Sandbox

parrot.org: Update on infrastructure

Andrew Whitworth: Reading a Line of Text

parrot.org: PACT: Infrastructure

parrot.org: The initial step

Andrew Whitworth: IO Rewrite Status

parrot.org: A few steps forward

Andrew Whitworth: IO Cleanups Status

Andrew Whitworth: IO Refactors

Andrew Whitworth: Destructors are Hard

and what still needs to be done.

recycle for destructors.

especially from our users.

parrot.org: GSoC starts today

Andrew Whitworth: Pending Branchwork

Already Merged

eval_pmc

remove_sub_flags

whiteknight/gc_finalize

whiteknight/gh_663

whiteknight/gh_610

whiteknight/io_cleanup1

whiteknight/sprintf_cleanup

Andrew Whitworth: Parrot 4.4.0 Banana Fanna Fo Ferret

parrot.org: 4.4.0 Parrot "Banana Fanna Fo Ferret" Released!

parrot.org: GSOC: Introduction

Andrew Whitworth: XML Is Hard

parrot.org: GSoC: Here We Go Again

parrot.org: LAPACK bindings for Parrot-Linear-Algebra

Andrew Whitworth: Various Updates

ParrotStore

Eval PMC

Sub Flags Cleanup

PCC

Rosella

Google Summer of Code

`io_cleanup1` Status

`eval_pmc`

`remove_sub_flags`

`whiteknight/gc_finalize`

`whiteknight/gh_663`

`whiteknight/gh_610`

`whiteknight/io_cleanup1`

`whiteknight/sprintf_cleanup`