If you're setting up a developer machine, you can start here: http://docs.mattermost.com/developer/developer-setup.html
If you're creating an install package you might check out this project: https://github.com/trashcan/mattermost-packager
Also, please consider joining the installers channel on the core team server.