Active Users:209 Time:28/03/2024 09:18:10 AM
i was *hilariously* immature in my 20s. love it! *NM* temeraire Send a noteboard - 21/01/2014 02:34:01 AM

View original postAs part of the fundraising drive last Dec (not last Dec, but last-last Dec... you know what I mean), I promised to import some of the old wotmania messageboards that Tor and I scraped from the site in its last few months.


View original postIt's taken me far, faaar longer than anticipated (approx 45 hours over the last two weeks alone), mostly due to the appalling state of the HTML I was trying to extract the data from. I also ended up needing to re-write most of the messageboard functionality to allow the archive posts to show properly.


View original postYou can view the archived boards in the new "Archive" section of the menu over on the left.






View original postIt's not possible to show the archive boards in the same way as the current ones, mostly due to the fact that there are no "recent" posts which can be displayed. Instead, the core homepage allows you to do some basic filtering of top-level threads, whereas the search page provide much more in-depth functionality to find specific posts.


View original postThere are a couple of things to watch out for:


View original post * The base filter form on the top of the board page won't allow a date range over more than 31 days. This is to stop the server from dying a fiery death as it tries to process a LOT of data. Even a full month's worth of posts take a good few seconds to load. Keeping date ranges small is a good way to keep the website running quickly.


View original post * If you have a premium account, you'll be able to use the "favourite post" functionality on the archived boards in the same way as you can on the normal boards.


View original post * I've added the ability for comments/new posts to be added to these old threads. The comment form is at the bottom of the page when viewing a post. It works in the same way as comments on quickpolls/journals/bugs/etc; just a single unthreaded list of comments for the whole thread. You can use the "order" option on the filter form to order threads by the number of "new" replies they have.


View original post * The way in which we saved the posts from wotmania means that on posts with over 100 responses, the sub-threading has been lost. The data I have allows me to identify the top-level responses within a particular thread, but not the sub-threading thereafter. These posts are instead shown with an ![exclamation](/site_media/images/messageboard/mbtips.gif) icon next to them, to show that it should be in a sub-thread.


View original post * The quality of Mike's HTML was terrible, and, even worse, was different depending on the contents of the post. This means that I needed to extract data in a number of different ways, depending on just how badly messed up the code was. The end result of this is that the formatting of many posts just isn't quite right. In particular, edit links show even though they don't work, post links are shown twice, and there tends to be a lot of extra space at the end of the post. Signatures are also poorly formatted, and tend to be a lot bigger than they should be.


View original postStill, it's good enough for now. If anyone's interested in a more indepth explanation of how we scraped the data in the first place, and how I went about extracting it and importing it into RAFO, let me know. It's all terribly exciting.


View original postAs a final note, you may notice that there is a big gap between the final post on CMB2 (14th Nov 2003) and the first post on CMB3 (9th Apr 2004). This is because, as per Mike's first post, the original CMB3 got corrupted and was wiped out.

"it's like the real world, except there's dragons!"

cw, you are missed...
Reply to message
Opening the archive: Community Messageboards 2 & 3 - 02/01/2014 03:59:21 PM 11332 Views
Awesome! Finally! - 02/01/2014 04:56:19 PM 1366 Views
Great stuff. - 02/01/2014 06:13:10 PM 1189 Views
Wow (not a world of warcraft reference) - 03/01/2014 12:46:16 AM 1219 Views
Wow redux - 03/01/2014 04:41:46 AM 1364 Views
If you can read this, the server is down. *pensive* - 03/01/2014 06:06:35 PM 1410 Views
Seeing a lot of those names is quite nostalgic. - 03/01/2014 09:09:56 PM 1243 Views
Thanks a lot for this! - 06/01/2014 08:03:38 AM 1226 Views
Woohoo - 11/01/2014 02:41:15 AM 1215 Views
Really really glad your message boards don't work at my work computer - 20/01/2014 03:04:37 PM 1128 Views
Man! I wrote some dumb crap when I was in my 20s - 11/01/2014 10:46:44 AM 1121 Views
Re: Opening the archive: Community Messageboards 2 & 3 - 20/01/2014 06:45:34 AM 1145 Views
i was *hilariously* immature in my 20s. love it! *NM* - 21/01/2014 02:34:01 AM 854 Views
Nice. - 21/01/2014 04:18:26 AM 1389 Views
*steals all your wse points* *NM* - 22/01/2014 08:16:32 PM 640 Views
HI TRIGGER *NM* - 24/01/2014 03:55:51 AM 688 Views
Ben, you are awesome. - 24/01/2014 03:57:24 AM 1123 Views
Now we get to relive Gems like this.. - 10/03/2014 11:45:14 AM 1136 Views
WOooOOooOOOooOWwwWwwww.... - 12/05/2014 01:20:00 PM 1130 Views
Before my time. - 21/10/2014 02:22:58 AM 969 Views
Re: Opening the archive: Community Messageboards 2 & 3 - 02/03/2014 05:42:50 PM 1569 Views
Oh Ben! What a wonder you are! ^_^ *NM* - 12/05/2014 01:16:23 PM 622 Views
Way Back Machine lets you see wotmania as it once was... - 02/09/2014 07:07:13 AM 1165 Views
You are amazing, thank you for all your hard work *NM* - 08/11/2014 08:12:04 PM 566 Views

Reply to Message