A Whirlwind Tour of Recurrent Neural Networks
•Descargar como PPTX, PDF•
1 recomendación•867 vistas
Recurrent Neural Networks (RNNs) can be used to generate text that will look like its original training data. There are many articles out there that show hilarious end-results of such adventures, but start-from-scratch walkthroughs that show the raw code, like this one, are hard to come by. This presentation will demonstrate what you need to join in, grab your own data set, process it, train it, and sample it. Training the data on a CPU can take hours, but in this session you can learn how training on a GPU with hardware acceleration takes only seconds. Come away from this session with your own datasets to “randomly” generate new bodies of text!
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (11)
Similar a A Whirlwind Tour of Recurrent Neural Networks
Similar a A Whirlwind Tour of Recurrent Neural Networks (20)
Más de Sarah Sexton
Más de Sarah Sexton (14)
Último
Último (20)
A Whirlwind Tour of Recurrent Neural Networks
- 1. A Whirlwind Tour of Recurrent Neural Networks Sarah Sexton Microsoft, Chicago Software Engineer @Saelia
- 2. Love building new things...? @Saelia
- 3. ...but hate thinking of a new name? @Saelia
- 4. Thinking of a video game genre is easy...
- 5. Run and train RNN To Run To pull or run Sarah’s pre-trained Docker snapshot to avoid waiting 8 hours, type: docker pull saelia/rnn-js To Train The way to actually make the RNN generate new Shakespeare text is with the data sampling script: th sample.lua -gpu -1 -checkpoint cv/checkpoint_12900.t7 -length 150 -temperature .7 GPU: Setting the flag gpu to -1 tells the code to train using CPU; otherwise it defaults to GPU 0. Checkpoints: While the model is training, it will periodically write checkpoint files to the cv folder. The frequency with which these checkpoints are written is controlled by the number of iterations, specified with the eval_val_every option. (E.g., if this is 1, then a checkpoint is written every iteration.) Length: An important flag is -length. 100 would generate a body of text 100 characters in length. The default is 2000. Temperature: An important parameter you may want to play with is -temperature, which takes a number in range (0 to 1, 0 not included), default = 1. Lower temperature will cause the model to make more “likely” but more boring and conservative predictions. Higher temperatures cause the model to take more chances and increase diversity of results, but at a cost of more mistakes. @Saelia
- 6. @Saelia
- 7. Learn: • Docker experience • RNN knowledge • Great names by AI @Saelia X1 X2 Y A1 A2 A3 A4 B1 B2 B3 B4 Input layer Output layerHidden layers
- 8. Superheroes Designed by Neural Network Speet Stank Red Fart Mister Man Rad Food Sapgirl Woop Ann Man Boomss Boark II Supperman Superbore Slonk Lid Man Green Hooter II Starm Surper Shartar Goons Nana Rider Farm Captain In Redink Wolver Man Wizler http://aiweirdness.com/post/140829108357/superheroes-designed-by-neural-network @Saelia
- 9. Quincelax • Abilities: Sturdy, Secene Grace • Hidden ability: Tunged Leus Tortabool • Ability: Healy Stream Strangy • Abilities: Wharmwbra, Darp • Hidden ability: Magic Guard Stangute • Ability: Banger • Hidden Ability: Drang Tyrnakine • Ability: Beak Eye Minma • Abilities: Buttery armor, Shell Armor • Hidden ability: Weak armor Pokémon Generated by Neural Network http://aiweirdness.com/post/147834883707/pokemon-generated-by-neural-network @Saelia
- 10. Recipes at your own risk! http://aiweirdness.com/post/163878889437/try-these-neural-network-generated- recipes-at-your
- 11. Craft beer names by RNN IPAs • Dang River • Yamquak • Bigly Bomb Session IPA • Binglezard Flack • Earth 2 Sanebus • Tower Of Ergelon • Juicy Dripple IPA • Wicked Geee • Yampy • Widee Banger Fripper IPA Strong Pale Ales • The Great Rebelgion • Thick Back • The Fraggerbar • Dankering • Third Maus • Sip’s The Stunks Belgian • Slambertangeriss • Devil’s Chard • Spore Of Gold • The Oldumbrett’s Ring • Gunder Of Traz • Cherry Boof Cornester • Humple Bobstore Barrel Aged Amber Ales • Snarging Red • Warmel Halce’s Comp Ale • Fire Pipe • Blangelfest • Stoodemfest • Ole Blood Whisk • Frog Trail Ale • Ricias Donkey Brain • Sacky Rover • Gate Rooster • Cramberhand • O’Brien Irish Red • River Smush Hoppy Amber Ale • Rivernillion Amber • Special North Imperial Red • Ambre O’Woo’s Omella Imperial Red Ale Stouts • The Moon • The Bopberry Stout • Cherry Coconut Mint Chocolate Stout • Black Morning • Sir Coffee • Shock State • Take Bean • Single Horde • Whata Stout • Shany Lace • Barrel Aged Chocolate Milksmoke • Shump http://aiweirdness.com/post/163753995072/craft-beer-names-invented-by-neural-network@Saelia
- 12. Harry Potter and the difference between word-level and character-le vel RNN http://aiweirdness.com/post/164291045392/harry-potter-and-the-word-level-recurrent-neural
- 13. A character-by-character, or “char” model takes one text file as input, and trains an RNN to predict the next character in a sequence. The RNN can then be used to generate text character by character that will look like the original training data.
- 14. New paint colors invented by neural network http://aiweirdness.com/post/160985569682/paint-colors-designed-by-neural-network-part-2
- 15. • Star Trek: The Next Generation
- 16. • Doctor Who and the Daleks!
- 18. • a fake lightning talk (generated from existing TED talks)
- 19. • The temperature flag makes the most difference. (Expects a number between 0 and 1.) • Changes the novelty and noise is the system, • Creates dramatically different output. • Lower temperatures (e.g. 0.2) makes the RNN more confident, but more conservative • It generates less noise, but less novel results. • Using -temperature 0.2 gives clear English, but includes a lot of repeated words. • Higher temperature makes more interesting/novel output, but more nonsense, misspelled words • Everything is a trade-off. • Experiment with all settings. Temperature
- 20. Temperature 0.3 and 0.4
- 21. Temperature 0.5 and 0.6
- 22. • There are lots of things that affect how well the algorithm does. Temperature adjusts: • whether the RNN always picks the most likely next character as it’s generating text, or whether it will go with something farther down the list. • Setting the temperature higher or lower can make the algorithm produce a much better output. Temperature 0.7 (my favorite)
- 23. Deep Learning Virtual Machine on Azure @Saelia
- 24. Deep Learning Virtual Machine on Azure @Saelia
- 25. Deep Learning Virtual Machine on Azure @Saelia
- 26. Deep Learning Virtual Machine on Azure: Price Calculator @Saelia
- 27. Deep Learning Virtual Machine on Azu re
- 28. • Commands to give you permission to write in Azure VM: • sudo chown –R username: /dsvm/tools/torch • sudo chmod –R u+w /dsvm/tools/torch/ • (replace “username” with your own username) Deep Learning Virtual Machine on Azure @Saelia
- 29. Deep Learning Virtual Machine on Azure @Saelia
- 30. Create your free account today! $200 credit to explore services for 30 days 12 months Always free of popular free services 25+ services aka.ms/MCTAzure @Saelia
- 31. Complete our survey for a chance to win a GoPro Hero 6! aka.ms/MCT18 @Saelia
- 32. Need Resources? Check out Microsoft Docs! Home of Microsoft Technical Documentation, API references, code examples, quickstarts, and tutorials for developers and IT professionals aka.ms/MCTDocs .NET ASP.NET SQL Enterprise Mobility +Security Dynamics 365 Azure Bot Service System Center Microsoft Education @Saelia
- 33. @Saelia Thank you! Ask me questions on Twitter: @Saelia Sarah Sexton Microsoft, Chicago Software Engineer
- 34. @Saelia Sequence Modelling and NLP With Deep Learning (Keras) Video: https://www.youtube.com/watch?v=ZmCzrPVzDQI Documentation Resources on RNNs: https://github.com/jcjohnson/torch-rnn/blob/master/doc/flags.md#training http://www.jeffreythompson.org/blog/2016/03/25/torch-rnn-mac-install/ https://github.com/jcjohnson/torch-rnn/issues/24 https://github.com/karpathy/char-rnn https://github.com/crisbal/docker-torch-rnn https://github.com/Element-Research/rnn https://github.com/zer0n/deepframeworks/blob/master/README.md
Notas del editor
- Hello everybody! Thank you for coming to my talk!
- I wanted to generate names for video games with an RNN. Recurrent Neural Networks are supposed to be this cutting-edge buzzword, the latest in Machine Learning Artificial Intelligence, a way to generate smart new text.
- There are lots of things that affect how well the algorithm does. One simple change turns out to be the “temperature” (think: creativity) variable, which adjusts whether the neural network always picks the most likely next character as it’s generating text, or whether it will go with something farther down the list. I had the temperature originally set pretty high, but it turns out that when I turn it down ever so slightly, the algorithm does a lot better. Not only do the names better match the colors, but it begins to reproduce color gradients that must have been in the original dataset all along. Colors tend to be grouped together in these gradients, so it shifts gradually from greens to browns to blues to yellows, etc. and does eventually cover the rainbow, not just beige.
- [examples of what people created, with our inspiration]
- [examples of what people created, with our inspiration]
- [examples of what people created, with our inspiration]
- [examples of what people created, with our inspiration]