When your brand new application looks shinny, it is time to put it on the Internet, process called deployment. Usually this involves several steps, like uploading application files, creating remote database, setting up configuration, etc.. Once it is setup and running you reazlise that your app is missing features X, Y and Z and deployment starts over and over again, this often leads to strange bugs.
I use simple lftp command line tool to automate project deployment via FTP and eliminate any possible deployment errors.
# The problem
When the project becomes complex, deployment is going to be complex too. It includes many steps and forgeting just one can lead to hard tracking situations. Likely you are going to catch bugs like: it works locally but does not on remote server.
Once it happens over and over again and there is room for an error, this needs to be automated, no more manual actions are needed. I want it just run-and-forget.
Here is the list of my basic requirements
- Mirror application code files, this involves only updating files that have been changed, project may contains thousands of files and this will take time
- Remove stale application files on server side which are no longer in use
- Skip uploading configuration files, This is a common case, overwriting database configuration on the server with local configuration.
- Skip uploading data folders, while some folders on the server may have user data in it, do not touch these folders
- Delete remote cache files, some frameworks and template engines create remote cache files, which needs to be removed
- Change remote permissions, to be sure folders can be used for uploaded files
- Fully automatic, setup once and run when needed
# Lftp - seems like a solution
lftp is a program that allows sophisticated ftp and http connections to other hosts. lftp can handle six file access methods - ftp, ftps, http, https, hftp. Every operation in lftp is reliable, that is any not fatal error is ignored and the operation is repeated. So if downloading breaks, it will be restarted from the point automatically.
lftp has shell-like command syntax allowing, it accepts on the input script like program. Each command tells lftp what to do. This gives perfect control of what is going to be uploaded or deleted.
Installing under Ubuntu is pretty straightforward: apt-get install lftp. There is also windows version available.
# Usage example
In this example, lftp script is used for uploading CakePHP project, which is not a simple upload task.
In the project folder I created file sync.sh which handles all the dirty job, below is the content of the file.
www is local folder. public_html is remote folder.
#!/bin/bash
lftp << EOF
set ftp:ssl-allow no
set ftp:passive-mode true
set ftp:list-options -a
open -u [USERNAME],[PASSWORD] ftp.host.com
# Upload site configuration
put -O public_html/dev/app/config/ www/app/config/bootstrap.php
put -O public_html/dev/app/config/ www/app/config/routes.php
# Mirror program code
mirror -R --delete --exclude config --exclude tmp --exclude webroot www/app public_html/dev/app
# Mirror webroot (not delete remote user files)
mirror -R --exclude user_data www/app/webroot public_html/dev/app/webroot
cd public_html/dev/app/tmp/cache/models
mrm *
cd public_html/dev/app/tmp/cache/persistent
mrm *
EOF
FTP username or password can be embedded into script itself, which is less secure. More secure way is to add additional lines of bash scripting to ask for password.
put - uploads one sinlge file. First argument is REMOTE file name, second argument is LOCAL file name.
mirror - synchronizes local and remote folder. Used swithes are:
- -R - sync from local to remote (uploading, NOT downloading)
- --delete - delete files on remote side which do not exist on local side
- --exclude - exclude directory by name
- cd - remote change directory command
- mrm - remove multiple files
Debugging
When lftp is run with "-d" switch it shows all FTP commands, which is good for debugging. One more good switch of mirror command is "--dry-run", it just shows what is going to be uploaded without any actual action performed.Beware! Do not do any experiments on live project with uncommitted changes!!! You may loose your changes. Do a project copy or commit all changes before.
# Conclusion
Now I am sure that after development round all changes will be uploaded to the server and nothing left behind. Upload script is easy customizable for future projects.