Review Blitz 2021: Post-Event Feedback

[K_S]

I'm...honestly not a fan of many of these suggested changes.

Would you be willing to elaborate I am curious?

Speaking as the 20 reviews 19k word effort writter here....

I am of mixed feelings on the scoring system as it stands.

On one hand I did cap myself at 20 with the mentality of "quality over quantity" with the idea of hoping to get two prizes I liked...

My goals were kitty attachment to my page and a picture. So goals met...

Honestly tho 20 was almost too much. Next blitz I am going to have to pull less and settle for less rewards... Because of that I am not a fan of thinning the prizes thats been tossed about. Hence my suggestion of donating unwanted prizes to a pile. Rises odds of the timestrapped to get something they want....

But on the other I really felt punished for the sheer effort I poped in getting overshadowed by what I saw were a bunch of smaller/mini reviews.

My mantra boiled down to "do what I can" and to not look at everyones scores because it was sheer frusteration after awhile. So I definitly think a "word count" system for reviews submitted to guage effort should be a thing. Perhaps one team to be judges of reviews alongside actually reviewing participants? Or even just a template that you can run your own review through and submit to the event...

As a reallt rd idea...

X words met .5 pts per bracket...
In depth analysis of
Character a .5 pts
Plot v. .5pts
Speculation cross reference of idea one to two... 1 pt
Ect....

Also I noticed a lot of shorter fics getting revisited over and over again. Postcards for example got a lot of attention... I personally avoided long multichapter fic myself. capping myself at two (luminary and stars) and I tried to draw attention to fics or authors that I thought werent getting much attention... Because while the stories coming up again and again were good ones certian stories just kept coming up and that didnt feel fair.

 

[IFBench]

...I'll just stay out of any future blitzes. Sorry.

 

[cynsh]

As a reallt rd idea...

X words met .5 pts per bracket...
In depth analysis of
Character a .5 pts
Plot v. .5pts
Speculation cross reference of idea one to two... 1 pt
Ect....

My concern with ideas like this is that a checkpoint-style list might look intimidating to people less confident about writing reviews. Plus it might force people to structure their reviews in ways that don't suit them, just for more points?

I do understand your frustration though at writing long reviews for few points, KS. Comes back to the scoring system I suppose. Taking the lower number of 'no. of chapters/words divided by 250' did frustrate me too at one point, for the same reason as you. I wonder if... just spitballing here, the two types of scoring could be combined, instead of having one or the other? E.g., writing 750 words for 4 chapters gives you 3+4=7 points. The prize thresholds would have to be adjusted significantly, of course. I dunno.

Also agree on your point about the RP-based prizes. As a fellow non-RPer, it would be nice if all RP-based prizes could be exchanged for something else.

Since I didn't say it anywhere else in the thread - massive massive MASSIVE props to Kint for organising, and everyone else involved in the blitz, especially the artists. You guys are incredible!

 

[IFBench]

Ok, I'll elaborate.

This all makes everything absurdly one-sided towards the people who write really long reviews.

As much as I'd like to write reviews like that, I can't. I physically can't. The most I've been able to reach is 1.5K words or so, and that was on 5 or so chapters. I struggle reaching that 250 word threshold as is, and people are talking about either raising that threshold or removing it entirely and basing points completely on wordcount only.

Aside from that, some of the suggested new point curves are really, REALLY harsh. Nubushi's suggested one would make it so that many people don't get their third spin until 70 points, which only the top 7 or so managed to reach. And with the weekly theme nerf, that's going to be much harder to achieve next time than it was this time. The "get three of the same prize type in a row and you get to try again" suggestion would only be able to apply to the very, very topmost reviewers, because they'd be the only ones to get three prize wheel spins. Everyone else would likely have two spins at best, if not only one.

All this combined would just make this a miserable event to partake in, and not one I want any part in. I'm already disappointed in my performance this blitz, I don't look forward to doing even worse next blitz.

 

[IFBench]

I wonder if... just spitballing here, the two types of scoring could be combined, instead of having one or the other? E.g., writing 750 words for 4 chapters gives you 3+4=7 points.

Then you could just write 250 words total on all of Hands of Creation or something and immediately get 70 or so points.

 

[Chibi Pika]

Gonna keep this short and sweet. I think any and all feedback on the metagame of the blitz comes down to one central question:

What is the intended outcome of the blitz?

That's not a self-explanatory question! I bet everyone will have a different answer! Personally, my preferred outcome is and will always be: incentivising more people to review who might not usually review much. So given that, my priorities are always going to be:

- Simple, easy-to-understand scoring (will reduce workload on the organizer as well)
- Low barrier to entry
- Accessibility

I think the comments that have been made about theme points being a bit disproportionate out of all earned points are fair. I still want the theme points to be easy to get and easy to track, but the ratio is maybe a bit skewed. I think the 250 word limit was relatively fair. I wouldn't mind lowering it if it would make things easier for less verbose readers, although it would have the unintended side effect of making long reviews balloon in points. Perhaps something like 200 as the minimum, with additional points for every 300 or 400 words beyond that. (Numbers not exact, just spitballing a concept that would allow long reviews to slowly increment more points while still lowering the barrier to entry.) Something with easy math would be nicest, for both participants and organizers.

Rewarding a review for covering more chapters would be nice, though of course we don't want 200 words on all of HoC to be worth 392875 points. There'd have to be some kind of limiter there. Maybe points per chapters vs points per wordcount can't exceed a certain ratio per review?

Example using the current 250 wordcount limit: a 2k review for one chapter being worth 9 points (8 for wordcount, 1 for chapters), but 2k review for eight chapters being worth 16 points (8 for wordcount, 8 for chapters.) But a 2k review for 80 chapters not being 88 points. Would it also be 16? Would it be 24? (Thus meaning that chapters:wordcount can't exceed 2:1)

Again, the exact numbers don't matter. It's more about the philosophy.

This might be unpopular to say, but I really want to incentivize people to read more content, even without necessarily writing longer reviews. As a writer, if I had to choose between a very long review on my first chapter, versus more people reading more chapters and getting further into the story, I would take the latter. Every single time. And tbh, as a reader I'd like to be incentivised to read further as well. Just getting myself to read a fic these days can be a struggle, let alone write a review.

Getting more people to engage with more stories is a more enjoyable outcome for me that meeting some platonic ideal of fairly rewarding high-effort reviews. But I know that's not true for everyone! That's why I think nailing down the intended outcome of the blitz is key.

 

[Shiny Phantump]

What if the points system was something more like "each review of a chapter (to a minimum of 200 words each) or each 300 words, whichever is greater.

I don't know if that's phrased clearly, but what I'm getting at is this: If you review multiple chapters, your words convert to points more efficiently than if you write one long review, the 251th word and more isn't completely meaningless. Though a little more complicated, I think the benefit of making longer reviews count for something while still benefiting multi-chapter reviews is worth it.

As for the weekly theme bonus part, I like the theme bonus part, but not the weekly part. There were a few times in this blitz where it occurred to me that I could withhold a review for more points, saving it to publish on a week it counted for. While I decided against doing so because it was against the spirit of the thing, it's absolutely incentivized under the current system and I got less points for choosing not to withhold them.

Here's what I would recommend: A list of theme challenges, each of which give +1 point the first time you meet the criteria, and only the first time, regardless of when that is. This does two things: It removes the incentive to not review things until they qualify, and it also limits the effect the challenges have on point totals. Once you've logged every challenge bonus, you can't get any more points from them, putting a hard limit on the amount of influence challenges can have on the scoring.

 

[IFBench]

A list of theme challenges, each of which give +1 point the first time you meet the criteria, and only the first time, regardless of when that is.

I really, REALLY don't like this.

 

[Dragonfree]

I really, REALLY don't like this.

Hmm, why? It's not very helpful to respond just saying "I don't like this" without elaboration - we can't discuss the issue and figure out something better if we don't know what's bothering you. By explaining your thoughts, we can work out improvements that everyone would be happier with, including you!

One thing I'd like to add, in general, is that by next year we might have the fic catalog I've been wanting to make up and running - a largely-automated searchable database of fics on the forums. In theory that should pretty easily be able to automatically pull in chapter word counts and such, making that easily accessible with minimal fuss. If particular features would be useful for managing the Review Blitz, I could definitely see what I can do about that. (Heck, potentially it could handle logging reviews, and the Blitz organizer(s) would just have to go through the day's submissions and verify.)

 

[Chibi Pika]

I don't know if that's phrased clearly, but what I'm getting at is this: If you review multiple chapters, your words convert to points more efficiently than if you write one long review, the 251th word and more isn't completely meaningless. Though a little more complicated, I think the benefit of making longer reviews count for something while still benefiting multi-chapter reviews is worth it.

I just wanted to say that I really enjoy the general philosophy behind this. Long reviews for minimal content still get something, but reading more content is incentivised.

 

[Flaze]

Okay so I actually don't have too many thoughts in regards to how the blitz can improve because I'm admittedly not that great when it comes to the technical stuff.

I will say that I had a lot of fun with the blitz, and while I maybe took a little too seriously, it was still worth it because it allowed me to read a lot and write a lot more reviews, which is something I'd been wanting to do last year since I joined the forums.

So on that note I really wanna give @kintsugi for organizing and managing the event since I know it was no easy feat and also to everyone that pitched in with prizes and reviews.

As for how the blitz can be improved. Honestly, at least in my personal opinion, I think it's more just a case of optimizing what's already there rather than making big overhauls.

I notice that there's a bit of a debate between whether the word count threshold should be longer or shorter. As someone that tends to write long reviews, I actually agree with the idea of lowering the wordcount. Not everyone can do long reviews for stories, especially since not all stories are written equally so the level of content one can comment on can also vary a lot, plus having a shorter wordcount means more reviews.

As to why I prefer this, well it's mostly cause while I know that the wordcount to review ratio can be annoying, especially if you literally spend whole days reading and reviewing like I did, I still did it because I wanted to. With long fics I split my reviews off in two chapters, meaning I literally only had to write 500 words and I would get a balanced ratio, but most of my reviews still ended up being far longer than that because I wanted to give all of my thoughts on what I was reading.

So what I'm getting at is that I didn't do long reviews for the points, I just did it cause I wanted to. At the same time, I do agree that maybe it's a little unfair for those that would like to get more value out of their long reviews if what they write ends up amounting to very little, especially with the time spent on it.

I also think that shortening the word count also means increasing the treshold for prizes because if we get the same level of activity...that'll be a lot of prizes, meaning it's a lot of work for the volunteers as well. So there's definetily a balance we have to strike there.

Either way, I think it's a result we can all come together.

Another thing I suggest is that once the blitz is close to starting maybe ther ecan also be a list of positions and roles that people can fill in to work through the blitz. If only because I feel like handling it all is too much work for just one person and I don't think kint should die for our sins :c

 

[IFBench]

It'd worsen the problem of people flocking to the same few fics for a theme. At least with the weekly themes, there's some variety in what fics people flock to. If people could only choose one fic for that, though, it's entirely possible and likely for almost everyone to flock to the same exact fic for a challenge and leave every other fic in the dust.

Plus, if there's not enough challenges, they'd be practically worthless. If there were only four challenges like this blitz had, that's only four points, which could easily be replaced by a single review. Even if there were something like 10 challenges, that's still not very much.

 

[K_S]

It'd worsen the problem of people flocking to the same few fics for a theme. At least with the weekly themes, there's some variety in what fics people flock to. If people could only choose one fic for that, though, it's entirely possible and likely for almost everyone to flock to the same exact fic for a challenge and leave every other fic in the dust.

Plus, if there's not enough challenges, they'd be practically worthless. If there were only four challenges like this blitz had, that's only four points, which could easily be replaced by a single review. Even if there were something like 10 challenges, that's still not very much.

We could lower entry for short poem fics

Apply a system like shiny suggested to tackle longer fics reviews.

And put a cap on fic used per contest bonus to diversify readers attention.

 

[cynsh]

Then you could just write 250 words total on all of Hands of Creation or something and immediately get 70 or so points.

I mean, that’s an extreme abstraction of the idea that’s obviously not right. It would need some adjusting, definitely. That said, @Shiny Phantump ‘s suggestion is better anyway :P

It'd worsen the problem of people flocking to the same few fics for a theme. At least with the weekly themes, there's some variety in what fics people flock to. If people could only choose one fic for that, though, it's entirely possible and likely for almost everyone to flock to the same exact fic for a challenge and leave every other fic in the dust.

Plus, if there's not enough challenges, they'd be practically worthless. If there were only four challenges like this blitz had, that's only four points, which could easily be replaced by a single review. Even if there were something like 10 challenges, that's still not very much.

I don’t recall there being a problem with certain stories being targeted for themes? I thought all the themes were pretty broad and allowed for a lot of choice.
I feel like 10 challenges is plenty - I didn’t write that many reviews in the whole blitz. And even if theme points don’t count for much, they still count for something. Helps people towards getting rewards and whatnot.

 

[WildBoots]

This all makes everything absurdly one-sided towards the people who write really long reviews

I wasn't sure if this was referring to proposed changes or the system we used this year, but I just want to point out that Pen was the winner kind of in spite of her long, "inefficient" reviews. She's got a really low ratio of points to word count. Most people who tend to write long reviews continued to do so for this event despite not getting any additional points for it.

 

[Shiny Phantump]

It'd worsen the problem of people flocking to the same few fics for a theme. At least with the weekly themes, there's some variety in what fics people flock to. If people could only choose one fic for that, though, it's entirely possible and likely for almost everyone to flock to the same exact fic for a challenge and leave every other fic in the dust.

First of all, more variety in challenges means that more fics will qualify, not fewer. Nothing about the weekly element encouraged each to review different fics from each other, it was "entirely possible" for us to all choose the same fics, but that just... doesn't happen. Likewise, people wouldn't choose just 10 fics to fulfill the 10 challenges, even though that is also "entirely possible."

It is occurring to me now that, if such a system was implemented, there should be a once challenge per review limit. That's how I was thinking of it, but I didn't seem to mention it. That means one fic that fulfils, say, 4 challenges won't become a lightning rod for reviews, since every review can only fill out one anyways.

In fact, a wider breadth of challenges mean that some can be more specific in order to draw attention to fics that would otherwise have gone unreviewed. With the weekly challenges, most of them were pretty straightforward and easy to fulfil. With a bigger set of simultaneous challenges, there could be some that are easy, like current ones, but for people who want the full list of challenges completed, there could be things like "review a oneshot" and "review the latest chapter of a fic with at least 5 chapters." Such challenges were too limiting to see use in the weekly themes, but the ability to have these kinds of challenges encourages people to branch out and try reviewing things that would otherwise have been left behind.

 

[K_S]

I wasn't sure if this was referring to proposed changes or the system we used this year, but I just want to point out that Pen was the winner kind of in spite of her long, "inefficient" reviews. She's got a really low ratio of points to word count. Most people who tend to write long reviews continued to do so for this event despite not getting any additional points for it.

Honestly it felt cheap not to put up the best reviews possible so thats why I just stuck with the long review personally.

Also you had to milk the point system near the end and do multichapter reviews to get the max out of it. I wound up not doing so and just tried to do a different authorneach time when i realized the shape and wheel of the system wasnt going to work with what I wanted to do.

 

[Umbramatic]

I feel like the event went great! I for one had a lot of fun despite initially being reluctant and was able to reach the prize thresholds I wanted despite being busy.

If I could make one suggestion... this will probably be unpopular, but I feel like we should ditch tracking wordcount completely and base points on actual chapters somehow. I always had about the same amount of words to say per review no matter how many chapters I read, and I ALWAYS had to stretch and bullshit to get to 250 words, so I didn't like the standards there very much at all. Plus if points were the lesser of two that means a 250 word review of 1 chapter got me the same number of points as a 250 word review of 5 chapters which. was kindea demotivating and led me to do the same old "reviews chapter 1 and that's it" stuff that makes me my own worst enemy.

If people are THAT averse to assigning extra points per chapter straight up (I know it does screw over oneshots) I feel like we should at least look into A solution based on chapter count and not wordcount. Because let's be real. I don't like emphasis on wordcount. And I don't like re-enacting my own pet peeves of "ten bazillion chapters on Chapter 1 and nothing else".

Also I prefer art prizes over anything else and am averse to pushing them up the latter? I do kinda like the idea of giving people a choice.

 

[Dragonfree]

Oh, just thought of one thing that was affecting me this Blitz--

The full extra bonus point for theme per review meant that it was always significantly better to review one chapter of a fic at a time - that meant the theme bonus effectively doubled the number of chapters I read, while doing extra chapters in one review meant losing out on one full point per extra chapter, and that definitely shifted my my behaviour here a bit. Just reducing the relative weight of the theme bonuses would help, but I would love it if next time, one way or another, the incentive structure made it at least as good to read and review multiple chapters at once (assuming you were to write substantially about each chapter either way) - I think that'd help combat that 1000-reviews-on-chapter-one syndrome.

 

[Shiny Phantump]

What if the points system was something more like "each review of a chapter (to a minimum of 200 words each) or each 300 words, whichever is greater.

On second thought, this is over complicated and I've thought a better way to do more is less the same thing. Or, two slightly different better ways. It not only is simpler, it also makes life easier for Kint. Making things easy for Kint is nice.

The change is making it so that you get one point per 200 words for each chapter, and then the excess words are 300 words per point. I can think of 2 ways to implement that:

The first and simplest way it to take the existing spreadsheet fields for total words written, and total chapters reviewed, and hooking up an automatic google sheets formula that sets points to (chapters + floor((total words - (200 * chapters))/300) + bonus). Players whose total average words/chapter over the entire blitz is <200 would need to be handled separately, but dealing with that edge case manually if it comes up is probably easier than automating it.

The advantages of that is being incredibly easy for Kint, doable with only the stats she already logs, as well making each word count. A 400 word review of one chapter doesn't 'waste' 200 words, so there's no incentive to pad out an extra 100 words to make it hit 2 points in and of itself, since those words still count towards total words written. It's also easy to log: We already said our chapters and total words, which is what would be needed.

It would mean that people not closely tracking their points might get a point more than they expected after some reviews. (For example, if someone wrote a 499 word review for 1 chapter, then later did a 201 word review for some other chapter, they may be surprised when the second review brings their points up by 2 instead of 1.) On the other hand, it'd always be one more than you'd otherwise get, so I don't think anyone would write into complain.

It is also technically exploitable in that someone who writes long reviews could put out a bunch of really short reviews to inflate their chapters reviewed count for more points, but if a bad actor who normally writes in-depth reviews started commenting "good fic plz update" on 20 different fics and trying to log it all for blitz, it's not going to be very hard to figure out what's up and intervene.

It would also be possible to apply the whole (chapters + floor((total words - (200 * chapters))/300)) thing to each individual review, instead of to the totals. iirc, Kint already tracks chapters/words for each individual review in the spreadsheet, and that means it's probably possible to hook up a google sheets formula to it. It also means that there's no edge case for reviews that are <200 words, because they aren't worth anything in the first place. I'm 90% sure it would still be harder to set up overall, though. (At least spreadsheet-wise.)

Edit: Fixed some typos erroneously suggesting hundreds of points per chapter