Log: Spoiler (13:00:45) ete: Aya: What's the current status of battle scripting? Does it all work properly, and does anyone have (even fairly basic) bots? (13:00:57) Luck>Skill: coyo's bot (13:00:59) Luck>Skill: works (13:01:00) +Aya: i've not really looked into it (13:01:04) Luck>Skill: it does random moves but it works (13:01:04) +Aya: but a couple of bots have been made (13:01:06) Luck>Skill: ! (13:01:13) ete: ok (13:01:15) +SteelEdges: ete (13:01:17) +SteelEdges: ask PokeWorldBW (13:01:19) +Aya: I think aerith has one on forums as well (13:01:20) +Aya: but idk (13:01:31) +SteelEdges: or go to the channel #Meow when he's on (13:01:39) ete: ok (13:02:06) ete: i think (13:02:10) ete: having a bot tourney (13:02:14) ete: could be really cool (13:02:34) ete: my mental sketch was three formats (13:02:50) ete: simplemon (like Shanai Cup) (13:03:22) ete: straight OU (5th gen OU, normal rules) (13:03:38) ete: and some kind of randomized mirror match (13:04:40) ete: if there was a fair bit of preparation time and some kind of prize (13:04:49) ete: i bet that would kickstart battle bots (13:05:54) ete: also, rules saying that after the tourney all bots must be opensourced (13:06:09) +Aya: i'd say someone needs to look at them beforehand at least (13:06:12) ete: then everyone can have battle bots :) (13:06:14) +Aya: so it's not just a player or something (13:06:22) ete: ideally (13:06:37) ete: the programmer would hand the script and team over to the judge (13:06:46) ete: and the judge would run the matches (13:06:53) +Aya: Yeah that would work (13:07:53) ete: hm (13:08:01) ete: who would be good to run it/may be interested? (13:08:27) ete: i was thinking of pinging the tourney admins, but i guess it's quite a different thing (13:09:32) +Professor Oak: The judge would require the technical prowess to understand how the bot works, and what it does. (13:09:45) +Professor Oak: So, ideally, someone capable of writing such a bot would be the best judge. (13:10:36) xdevo: moogle (13:10:38) xdevo: ! (13:10:52) +Aya: hi (13:11:01) +Galblade: nomnomnom (13:11:13) ete: hm, if the bots just run, then they probably don't need to understand. but they do need to be someone compident (13:11:27) ete: and, yes, moogle would be good if she's interested (13:11:57) +Aya: Well it depends on whether or not I can be bothered ever writing a bot. If I do, then I can't judge, but otherwise I'm fine with it (13:12:26) ete: you could still judge matches between other people's bots? (13:12:39) +Aya: Well that's true (13:13:43) ete: i guess one issue (13:13:53) ete: is people coding on their own computer (13:14:01) ete: which is slower or faster (13:14:14) ete: may make less than ideal arrangements to avoid timeout (13:14:52) ete: we'd probably want to publish the speed of the judges computer (13:15:02) ete: to at least give them an idea[/HIDE] Basic idea: Let's make a bot tournament. I would like to see what people can come up with, AI is interesting. Things to bear in mind/discuss Someone other than me should run this. My activity is not consistent and I'm not a scripter so poor choice for keeping a close eye on things. Need a way to prove a bot is not just a human - solve by passing code+team to a judge and have them run it Judge's computer may be a difference speed from programmers - not entirely solvable, but publishing the judges computer specs will help Moogle may well be judge. Unless she's making a bot. Having a prize would be very positive. coyo, up for putting a bit of ad money in as a prize? Maybe 30-50 euros for winner, 10-20 for runner up? Bots can play best of <large number> games easily. We could do something like BO7 from the start, with finals being BO 21, and pick some matches to showcase as finals. Round structure depends greatly on number of participants. With a fairly low number everyone plays everyone is probably best, larger maybe separate to groups of 4 and have top 2 from each group progress? Or round robin is an option. Letting programmers review/edit their code/teams between rounds based on how they worked or not? Got to give a fair amount of warning to get good bots, perhaps a month? Restrictions on the bot, I think perhaps using information gained about an opponent and their team from previous battles should probably be banned, to simulate the lack of information of a normal new match. Start the bot in the same state for every battle. Ban on any bot talk designed to interfere with other bots (e.g. spamming to lower connection speed of a bot on low time, trying to mimic messages) No communication effecting battle behavior between the bot and users, so the programmer can't control it via posting messages or anything like that Multiple/randomized teams within a round should be allowed imo. Means a round is not so easily decided by team advantage, and rewards programmers who make a bot capable of playing with different styles. Handling time limits, use the default clock and leave it to programmers to make sure they include checks to avoid timeout (if seconds_remaining<10 select random, weee)? All bots and teams would be opensourced at end of tournament, downloads for them published, and a channel opened (#AI, I'll be hanging out there when I can) which would have all the bots available for challenge. Opensourcing during preparations also encouraged. Aim of tournament is to stimulate AI improvement, putting everything in public helps that. The judge would probably want a framework for helping automate battles and recording results. Crashed/frozen bot=loss of game, but not entire round. Allowing the script to call other programs/non-userscript stuff? Probably should be allowed, if someone wants to write something in C++ for speed then that should be fine. Possible security concerns for the judge though. Allowing programmers to enter multiple bots? Personally, if they were significantly different, I'd be okay with this. Making several bots with different styles would also be a good way to test out the bots. Maybe limit to two/three, and require explanation of how they were different/permission from a judge who'd check they were not clones with some minor tweaks. What rules to use for actual games. My initial thoughts were three (simplemon (like Shanai Cup), straight OU (5th gen OU, normal rules), and some kind of random sets mirror match), but maybe that splits attention too much and it'd be better to focus on just one area. Probably straight OU? Obviously, much to be decided. Thoughts please?
Short Additional ideas: The tournament will take 24 hours, thus no timezone problems. (It shouldn't be to difficult for programmers to let the computer run for a night or two) At 5 seconds per move (hey it's a computer)- 5 minutes (60 moves) per match about 288 rounds are possible. (With the constraint of one battle per bot at a time) 20 bots (probably way to much) equals 14 battles against each opponent. 2660 battles in total. (Numbers may vary, I'm not an active player) -- Needs development. I don't think a human will want to play consistently 288 battles, 24 hours...The room will be open a 24 hours before and after. There may be a testround a week before. Alternatively: A permanent testroom for everybody to play against bots. All battles will be recorded and released. Thus no real judges(See also next)CPU power will be donated by the programmers. Pro: No environments need to be set up: cheaper, safer, easier. Con: Open source can't be enforced. But still encouraged. Could be less fun for repeated contests: copied bots.Once a room is created, monthly contests may be kept. I think the quality will go up after two or three contests. The tiers may be rotated, every month another tier. This will keep improving the AI's, and someone can focus on a specific tier with one battle every three months or so. Multiple teams should be allowed (imo), this counters the learning algorithm (which probably would be to hard to implement by the first contest). But will allow some of the more interesting AI's. If multiple people use learning AI's this could generate nice metagames. This is also easy to implement, there are enough public teams. This again removes the need of a artificial constraint while doing less work :). Practical implementation: You can create a seperate channel with a bot(Manager) that kicks everybody unregistered that tries to start an unsanctioned battle. Alternative: Ignore all unrelated battles. -- How to make sure that somebody doesn't tie up a bot with fake battles? Alternative: Allow multiple battles at the same time. (The low level protocol supports this, Javascript probably also, needs development in the client(or I didn't find the correct combination yet)) This also means more battles could be played (who doesn't have a dual core?) Alternative: Only allow challenges by certain people (~IRC flags) Alternative: All bots only accept and send sanctioned battles. (Easiest, and it's in the interest of all bots to adhere to the rules)The Manager has a list of registered bots. You can log with your bot and let it play. The accounts are created on a forum, perhaps registered by a trusted person. The Manager has a command that orders one bot to challenge another or supplies a list of all battles a bot still needs to play(slower). The Manager listens to all results. A bot may disconnect, needs to reconnect and continue the current battle. At the end some missed battle can be played. Leftovers are drawn.
There are four main reasons I'd prefer to go with the judge model, first it makes any cheating via human assistance impossible, second.. fairness. I'm not all that keen on letting users who have a really good computer (or in the extreme those who hook their university's network up to the bot) have a large advantage over those with an older comp. It also removes the whole finding your opponent part, which often holds up tournaments, and not everyone can/will leave bots on all the time. Last consideration is it forces open source. Five seconds seems likely to be too restrictive, and far too uniform. For a game as complex and branching as Pokémon, it takes some time to work through possibilities, and that time varies with the situation. Using the same clock as human players may reduce the number of matches playable, but it will make each of those matches a much higher quality and make bot writers optimize towards time constraints they'll use when fighting humans. Various future considerations (rotating tiers) are possible. And the practical implementation section kind of shows up all the complexities introduced by not using the judge model, which can be bypassed entirely.
A judge model will indeed be easier to enforce the various rules set. But I think it will be really hard to be faisible. I don't have access to (multiple) servers. Considering that all bots will take 100% CPU power, every bot should have a single processor available. I don't know how much turns an average battle takes but using 4 hours of CPU-power just to play one game(24 turns 5 minutes each(perhaps not 60 turns this time :))... A single elimination tournamen with 20 bots will take minimum 2*10 hours and about 1.5 CPU days (which I think would be really long). I think this is the biggest issue, if there is a need to set up servers this will be a large cost. These servers need to be available every tournament. Note: Round robin will take the 19*2 hours and about 20*19*2/2 (=380) CPU hours. The judge model will also restrict us to a single language (javascript). This is not neccesary bad, but should be taken into considiration. Low level languages (C, C++, Java, ...) would be to easy to break out of. Also there may be a large performance difference. Creating a safe enviroment for them will be to hard. Also there is need for a clear API where for implementing a bot and enforcing the necessairy rules. (This will probably be a big plus, the entrycost would be less.) 5 seconds would indeed be to uniform. I would set a fixed total match time, 5 minutes each, like a blitz game. But then one could take advantage of forcing the opponent to make more choices, which would be a bad effect. This won't indeed be a practical solution. I suppose the complexity of both implementations will be about the same. We still need to be able to load the bots, and order a fight. For the judge implementation an API must be developed also, which will reach a lot more programmers(+), otherwise an API is optional. P.S. what would be a practical number of bots, number of matches between bots and what is the average number of turns for a batte? These are keys in the scalablility of the problem...