What VN jam statistics tell us.

Precise numbers, blurry meanings.

I don’t quite know how I got to this point, especially this “deep”, if you will, but what started as a simple “infographic” in 2017 about how many NaNoRenO VNs were made that year later grew into me monitoring several VN jams on itch.io, collecting statistics (such as number of creators, people who entered, when finished VNs were published, or how “finished” they were), reflecting them in graphs and trying to find patterns.

It’s both good and bad news that I, at the moment, have arrived at the conclusion that no useful information or accurate predictions can be derived from this data. Like art I suppose, they don’t really want to conform to any patterns, and each shape they produce is somehow unique. Even the ones that look the same all have different flavors, different circumstances underneath them, as if I were looking at a set of small flowers — all alike, but all made from different materials. And that is even before any nitpicking is invited. It’s a good thing, I feel, but as a way of reflection on this time, I’d like to highlight some interesting observations and theories I encountered on the journey that led me here.

Starting with what data to collect. Namely the number of “Participants” and “Submissions”.

At its core, it should be incredibly simple. A jam on itch is set up, and a someone signs up. The count for participants goes up by 1. After they are finished with their VN, they publish it to the jam page, and the count for submissions goes up by 1 as well. A nice, 1-to-1 ratio.

Then, a variation: two people signed up, and the second one didn’t finish. Then it follows that the jam has had 2 participants, one of them finished, and the other one did not.

These statistics are relatively easy to collect from itch, and in theory should be the most powerful as well, especially for predictions. If we determine that VN jams have a general 25% completion rate, and take that as a basic rule, we could predict that a jam with 100 participants would produce 25 VNs. And indeed in the first few jams I picked for the data collection, this simple metric seemed to be fairly accurate. If you took the number of participants, and took about 20% of that number, you’d get the rough number of submissions at the end of the jam.

Looks similar — but it isn’t.

But even though these numbers happen to work out, it was quite obvious that their underlying structure was subject to several problems, such as:

1. Multiple people signing up and submitting a single VN together (teams): Since itch counts the number of submissions (even if they have multiple authors) rather than the number of participants when it comes to finished works, this results in formally more people “not finishing”. For especially large teams of sometimes tangentially involved participants, this can generate large discrepancies, if the team doesn’t create a special itch user that is “their team”.
2. People submitting multiple VNs to a single jam: In case people submit two and more VNs for a single jam, the outcome is that the results will be skewed towards more participants appearing to have “finished”. This is a more rare occurrence, but it does happen.
3. Collaborators who haven’t joined the jam: Sometimes people participating in the jam don’t have an itch account and/or don’t join the jam with their itch account if they are not the main author, and are just credited on the game page. Sometimes they are one of the authors (itch allows users to co-own entries) but do not formally join the jam. This way they cannot be counted towards the number of actually active participants which makes the numbers less accurately reflect the reality of how many contributors participated.

For these reasons, and more, it will inevitably be revealed that the above “20%-of-participants” patterns were more of a coincidence than anything else. There will be some rudimentary correlation, but none so strong that it can be used to make any meaningfully accurate predictions or conclusions. And that will apply to statistics between the same jams as well, especially ones as broad as NaNoRenO, for example.

Do you (want to) see a pattern?

And then, there are the jams themselves.

They are just too different. Even among those that are strict in only allowing visual novels still span a wide variety of concepts and rules, so it’s difficult to even say what a “VN jam” actually is. Here are some of the factors contributing to the heterogeneous nature of VN jams:

1. Time limits: Some jams are a week long, some last for 2 months. The effort, approach and output of them will inevitably be different. Participants of very short jams producing smaller VNs will be ill advised to use a large pool of collaborators, while in long-lasting jams the fluctuations in terms of project cancelations, and team members will be greater, to name just on effect on the number of participants ultimately recorded by itch once the jam is over.
2. Content creation: Some of the most significant rules for VN jams apply to how the individual jams handle creating content for their entries. For example, can you start work prior to the jam? This seriously impacts commitment and affects who can participate. Can you use a project that hasn’t specifically been envisioned for this jam? This will have an effect on the complexity of the output as more elaborate VNs can be associated with the jam.
3. Cross-jamming: Can you even take a work you’ve been doing for one jam and which also happens to fit into another one and submit it there (as we now have jams running parallel or overlapping one another’s time frames)? And if yes, should they count separately for each jam, or only be assigned to one jam (based on intent, perhaps)? Some jams allow certain types of cross-posting, some don’t, and it will be difficult to reflect this accurately. With the right set of jams aligning for a dev, it is possible to submit a single game’s initial demo to one jam, its partially completed “Part 1/Chapter 1” version to another, the full version to yet another, and then an extended/reworked version with extra features and routes to a fourth one, if they happen to fit themes and other specifications as the game’s development progresses and changes its form and proportional composition of elements and themes.
4. Partial works: As mentioned above, it’s also about how “finished” your VN needs to be. Many jams allow demos, partial releases and unfinished works in general. This way the focus is shifted more towards their state of being “submitted” rather than “finished”, all the while other jams demand finished works, or reasonably complete experiences, bringing those terms closet together again. However in all cases the VNs end up being counted as a single submission, ultimately widening the definition of what “submission” means.
5. Extent/scale of works: There are big differences even within a jam in terms of the extent and scale of the VNs themselves. For NaNoRenO a simple 5-minute VN done in half a day is counted the same as a 30.000-word 10-people production that took the entirety of the month and pushed the team to their limits, and which (some jams will allows this) can also be commercial or at least “pay-what-you-want”. In both cases (“big or small”) it’s a single VN, counted towards the total as a single digit, even if we would find the work and value equivalent of 20 or more such small works in that other hypothetical larger one.

The form-restricted, short and finished-only O2A2 VN jam is a particularly clear example of breaking patterns when compared to month+ long jams allowing partial submissions and significant form freedom.

Additionally, the ability to update your VN page — going from demo, to full release and extended version, and replacing any original files in the process can (years later) give a false impression of the jam, making it seem that richer and more complete games have been created than was the case at the time the jam ended — at the very least at a glance. Also, if the original submissions have been removed, it becomes a labor-intensive or impossible task to figure out the original version lineup submitted to the jam.

The list of such realizations is long, so let us move on to some examples of the diversity of participants themselves.

Of course, not all participants are the same, in fact there are many motivations for people to join a jam and to release a game within a jam. Some want to do it for fun, some want to jumpstart a longer project, and some want to use jams to increase visibility of their work. Some people will join a jam without it feeling like a commitment, and some will only formally join a jam if they have something to publish (a “drive-by”, if you will, where they will join and submit at the same time). Speaking of which, if such people don’t make the deadline, they may have gone through the whole jam participating with all their heart and there will not be a trace of this reflected in the statistics.

The number of participants is also an interesting area to think about more deeply. Some jams have 20–40 participants, and even though percentages may suggest a pattern when compared to similar percentages from 200–400 people jams, it’s likely just coincidence. A few people in those smaller jams behaving differently will have a large impact on percentages in small jams, but actually that doesn’t mean that 200–400 people jams are much better suited to draw meaningful conclusions. Taken into account all the limiting factors (rules) alone, likely not even 500 participants is enough to produce a reasonably uniform pool from which patterns applicable to similar jams could be extracted. And 500 is — at the time of writing this a number reserved for the “juggernauts” only.

All of this for me ultimately results in the realization that no useful practical application that can be relied upon can come from any of these graphs and charts.

On two occasions I have inaccurately predicted results of VN jams based on the data I had collected from previous jams. They were of the simplest “submissions are 20% of participants” kind, and still they were off enough to be no better than a guess.

I could still present some other “evaluations” that the data shows, such as the numbers of how many VNs were released which day of the jam, but why do it? Most VNs get released in the last 24 hours of the jam, which just means people want to use their time to the maximum. That is, frankly, neither surprising, nor useful.

Most people hand in their taxes, university papers, and release jam VNs at the very end of their deadlines.

So what’s left? Speculating that given a number of participants (if they are in the range of 10 to around 700), we can typically expect about 10%-60% of that number to be the number of submissions of various lengths and states of completion, and that the majority of such releases will happen at the very end of the jam? That is so vague, it’s effectively equivalent to no prediction at all.

—

I like statistics — I like playing with them, which is probably why I stuck with this for so long. But I do want to acknowledge that without practical use it’s merely a fun exercise, serving mainly to satisfy the human need for pattern recognition. Retrospectively quantifying the past is of no use, if each new event is so different that we can learn only the most general of rules — such as “give yourself enough buffer”. That can be achieved by simple observation, or experience, and crucially isn’t anything specific to visual novel jams.

It’s not a bad thing, necessarily, though. Jams (and VN jams) are unpredictable, and the spontaneous energy is part of their charm. Also, there is a certain level of satisfaction to be had from the fact that you can look forward to getting a different experience from something as specific as the world of “VN jams”. Look at any of the single jams’ entries — the variety of styles, genres and lengths is considerable, and that is already despite some elements being pre-defined by the jam’s restrictions. More than anything it just seems to show just how unique the approaches, styles, and stories are that VN makers create.

In fact, that may be the only truth I was able to find.