Browsing Category "Infrastructure"
14 Nov
2010
Posted in: Django, Infrastructure, Python
By    No Comments

Django with virtualenv on Webfaction

I really like the Django web framework and I like virtualenv enabling me to use different python configs and packages for different projects. And I have to say I like Webfaction, so easy, so good! (No, I don't get paid by them) This is a quick guide on how to setup Django in virtualenv using mod_wsgi with a Webfaction account.

  • Create an "mod_wsgi" application and create a website to use this application
  • connect to your accounts shell using ssh
  • enable Python 2.6.x as default python (as written in the Webfaction documentation)
    • run "vi ~/.bash_profile"
    • append the line "alias python=python2.6" (navigate to the last line, push "i" for insertion, write the line, push escape)
    • save the file and exit (enter ":wq")
    • reload the bash profile "source ~/.bash_profile"
    • ensure it worked by checking the Python version "python -V"
  • install "pip" and "virtualenv" (be sure to use easy_install-2.6 to use the setuptools of Python 2.6.x):
    • enter "easy_install-2.6 -U pip"
    • enter "easy_install-2.6 -U virtualenv"
  • I like yolk, that enables you to list the installed Python packages, install it using pip if you like
    • "pip install yolk"
  • Note that these packages (pip, virtualenv, yolk) are installed in your global Python. I recommend to not install any more packages to the global installation as virtualenv enables use to install all packages we need into the virtualenv Python lib.
  • navigate to your recently create mod_wsgi webapp, there should be 2 folders, "apache2" and "htdocs"
  • create the virtualenv for your project in the wenapp folder, name it like you want, I'll use "ve" here
    • "virtualenv --no-site-packages --distribute ve"
  • a new folder "ve" with the virtualenv was created. Please refer to the virualenv documentation for more information and usage of virtualenv.
  • Now let's install yolk into the virtualenv to see which packages are in there
    • either activate the virtualenv and install:
      • "source ve/bin/activate"
      • "pip install yolk"
    • or use pip magic to install into an virtualenv without activating it:
      • "pip -E ve install yolk"
  • now you activate the virtualenv and execute yolk:
    • "source ve/bin/activate"
    • "yolk -l"
    • "deactivate"
  • Install Django and any requirements the same way as demonstrated for yolk.
  • Assuming you installed Django into the virtualenv and created a Django project, we now must adjust the Apache config
    • backup the httpd.conf from /webapps/<yourdjangoapp>/apache2/conf
    • create a django.wsgi file somewhere in your webapp dirdctory (I use the conf folder of apache)
    • scan the httpd.conf for "Listen XXX", XXX ist the webapp port
    • remove the <Directory>...</Directory> segment
    • append the following telling the Apache to use the wsgi config we create in the next step:

NameVirtualHost *:<webapp-port>

<VirtualHost *:<webapp-port>>
ServerName <SomeServerName>
WSGIScriptAlias / /home/<your-username>/webapps/<your-webapp-name>/apache2/conf/django.wsgi
</VirtualHost>

  • Edit the django.wsgi file:

#!/usr/bin/python
 
import os, sys, site
 
# add virtualenv python libs
site.addsitedir('/home/<your-username>/webapps/<your-webapp-name>/ve/lib/python2.6/site-packages')
 
# append the project path to system path
sys.path.append(/home/<your-username>/webapps/<your-webapp-name>/ve/')
sys.path.append('/home/<your-username>/webapps/<your-webapp-name>/ve/<your-django-project-name>')
 
# set the settings module
os.environ['DJANGO_SETTINGS_MODULE'] = '<your-django-project-name>.settings'
 
# init the wsgi handler
from django.core.handlers.wsgi import WSGIHandler
application = WSGIHandler()

  • Restart the Apache (/home/<your-username>/webappas/<your-webapp-name>/apache2/bin/restart) and everything should be fine!
24 Oct
2010
Posted in: Backup, Infrastructure
By    No Comments

Simple MySQL Backup @ Github

A while ago I wrote about a little script I created, for simple backups of MySQL databases and emailing the backups.

I found some time to continue on the project and moved it to Github. You can find it as well as documentation there!

Simple MySQL Backup @ Github

27 Sep
2009
Posted in: Backup, Database, Infrastructure, MySQL, PHP, Programming
By    1 Comment

Simple MySQL Backup at Google Code

A while ago I wrote an article about a script I created to backup databases. Since then I had to make little changes and decided to put it to Google code as Open-Source-Project and released the new version 0.1.1 with some minor changes.

If you're interested in the script, head over to its project page and check the appropriate pages like the Changelog and the Installation and Requirements page. Feel free to download and comment  it.

UPDATE: You can find the project now on github!

12 Jul
2009
Posted in: Bugtracker, Infrastructure
By    No Comments

The Bug Genie 2 SVN Integration on Windows with VisualSVN Server

In my local development environment at home I use The Bug Genie 2 for bugtracking. The tool is quiet cool, although the german translation is totally broken (I fixed it and send it to them, let's see what happens). It comes with a module called "svn_integration" for integrating SVN into the tracker to automatically have updated issues when the SVN comments contain special keywords. That's quiet cool, too, but unfortunatly does not work for me.

I'm using Windows Vista Business x64 as os and for SVN the wonderful, free and easy VisualSVN Server. Next ugly thing on The Bug Genie is that there is no documentation for the modules. I found an entry in the forums how to use the integration in windows, but that did not work out of the box. But the code is already within the module, what is good.

So open modules/svn_integration/post_commit.php from your buggenie installation directory and adjust this line with your path to the bugtracker installation dir:

define('BUGS2_INCLUDE_PATH', 'D:\\xampp\\htdocs\\bugs\\');

Then create a batch file in the same directory called post-commit.bat. You can see the source below, copy it and adjust the following variables. I used the code from the forum post and adjusted it a little. Remember that urls with whitespaces do not work, create a symlink or rename your directories:

  • Path to svnlook.exe (you see my VisualSVN Server resides in D:\VisualSVN_Server, the default location C:\Program Files (x86)\VisualSVN Server does not work because of the whitespaces) VisualSVN Server has all the svn tools in its bin directory.
SET SVNLOOK=D:\VisualSVN_Server\bin\svnlook.exe
  • The path to your php executable. I used XAMPP, so it is in D:\xampp\php\php.exe if installed in root on drive D.
SET PHP=D:\xampp\php\php.exe
  • The path to your SVN directory. This is where svn stores its data.
SET SVN_PATH=D:\svn\
  • The post-commit hook we create later takes two arguments from VisualSVN Server, the path and the revision. Unfortunatly VisualSVN Server uses the url as path which collides with the svnlook command we use to determine if a bug was mentioned in a revision comment, as svnlook wants the real file path.
    So we have to adjust the url to a path. As there is no real substring functionality for windows batch (or I did not search long enough) we simply count the length of our SVN url. In my case VisualSVN Server listens at https://amanda:8443/svn/, so a repository would be at https://amanda:8443/svn/my_repos. The server root (https://amanda:8443/svn/) has a length length of 24. Change the 24 to the length of your url. The batch then combines the SVN_PATH and your repository path to the local path, D:\svn\myrepos in my case here.
SET REPOS=%REPOS:~24%
  • Once again, set the path to your installation directory of the bugtracker (replace D:\xampp\htdocs\bugs with your path):
%PHP% -f "D:\xampp\htdocs\bugs\modules\svn_integration\post-commit.php" "%AUTHOR%" "%REV%" "%COMMIT_MSG%" "%CHANGED%"

After your saved the file, open VisualSVN Server and the properties of your repository. Select the the Hooks tab and doubleclick the Post-commit hook. Enter the path to the batch file with the 2 arguments path and revision, in my case I entered "D:\xampp\htdocs\bugs\modules\svn_integration\post-commit.bat %1 %2", where %1 is the repository path and %2 the revision number.

Finally you have to install either WebSVN (which I prefer) or ViewVC and specify the path to your repository (in my case it is http://localhost/websvn/listing.php?repname=my_repos&) in The Bug Genie module settings for svn_integration.

One final and important hint: your user in the bugtracker has to have the same username as in SVN. By default, the first user in the tracker has the username "Administrator". My SVN user does not have the name "Administrator", so I logged out of the tracker, changed the property "uname" of the MySQL table "bugs2_users" for the admin to my SVN username, logged in and then everything worked fine.

As always, feel free to add comments and spread the word

And here the post-commit.bat code:

@echo off
SET REV=%2
SET REPOS=%1

REM path to svnlook (remember that paths with whitespace do not work e.g. C:\Program Files --> use mklink to create a path without)
SET SVNLOOK=D:\VisualSVN_Server\bin\svnlook.exe

REM path to php executable
SET PHP=D:\xampp\php\php.exe

REM path to svn (not url!)
SET SVN_PATH=D:\svn\

REM maps the svn url to the svn path - straightforward but the easiest way
REM replace with the length of your svn url e.g. 24 for https://amanda:8443/svn/
SET REPOS=%REPOS:~24%

SET REPOS=%SVN_PATH%%REPOS:/=\%

%SVNLOOK% log -r %REV% "%REPOS%" > COMMIT_MSG
REM SET THE COMMIT_MSG FROM THE FILE. The file is expected to contain only one line with this value
FOR /F "delims=" %%A IN (COMMIT_MSG) DO SET COMMIT_MSG=%%A
echo COMMIT_MSG=%COMMIT_MSG%

%SVNLOOK% changed -r %REV% "%REPOS%" > CHANGED
REM SET THE CHANGED FROM THE FILE. The file is expected to contain only one line with this value
FOR /F %%A IN (CHANGED) DO SET CHANGED=%%A
echo COMMIT_MSG=%CHANGED%

%SVNLOOK% author -r %REV% "%REPOS%" > AUTHOR
REM SET THE AUTHOR FROM THE FILE. The file is expected to contain only one line with this value
FOR /F %%A IN (AUTHOR) DO SET AUTHOR=%%A
echo COMMIT_MSG=%AUTHOR%

%PHP% -r "echo urlencode($argv[1]);" "%COMMIT_MSG%" > URL_COMMIT_MSG
REM SET THE URL_COMMIT_MSG FROM THE FILE. The file is expected to contain only one line with this value
FOR /F %%A IN (URL_COMMIT_MSG) DO SET URL_COMMIT_MSG=%%A
echo COMMIT_MSG=%URL_COMMIT_MSG%

%PHP% -f "D:\xampp\htdocs\bugs\modules\svn_integration\post-commit.php" "%AUTHOR%" "%REV%" "%COMMIT_MSG%" "%CHANGED%"
27 Jun
2009
Posted in: Backup, Database, Infrastructure, MySQL, PHP, Programming
By    6 Comments

MySQL Backup Skript mit Emailversand.

  • english
  • german

A while ago I found a good script for backing up a MySQL database and sending it via email to a recipient here.

The script is cool, but I didn't like its structure and the fact, that you have to add the database values inline and that it can only backup a single database. For this reason I rewrote it a little and you can download it here.

Features:

  • Backup of mutliple databases
  • Sending of backups to multiple users

For every single database a mail is send and there is no file saved on the server. And wow: it's really easy to configure!

How do I use it?

  • Download the current version (0.1)
  • Adjust the backup.php (you're getting help by my wonderful comments)
  • upload everything to a directory of your choice
  • if applicable, create a cronjob to periodically execute the script

And here the backup.php that calls the appropriate classes and executes the backup (also included in the download):

ini_set('error_reporting', E_ALL);
// include the files
require_once 'MySQLConfig.php';
require_once 'MySQLBackup.php';
 
// add some databases to backup
// the domain will be appended to the email subject and is also included within the sql file for identification.
$cfgHost0 = new MySQLConfig('username0', 'password0', 'database_name0', 'domain0');
$cfgHost1 = new MySQLConfig('username1', 'password1', 'database_name1', 'domain1');
 
$backup = new MySQLBackup();
// the path to the directory where this script is resided
$backup-&gt;setExecutionPath('/srv/domain/backup/');
// add the database configs to backup
$backup-&gt;addDatabaseToBackup($cfgHost0);
$backup-&gt;addDatabaseToBackup($cfgHost1);
// the sender of the backup mail
$backup-&gt;setSender('admin@yourdomain.com');
// add some people to receive the backup
$backup-&gt;addRecipient('john@yourdomain.com');
$backup-&gt;addRecipient('frank@yourdomain.com');
// execute the whole thing
$backup-&gt;backup();

If there are any problems or suggestions or feature wishes, please comment this post - thanks!

PS: to create a cronjob log into your server using SSH, then execute "crontab -e" to edit the crontab and insert for example "0 2 * * 0,3 wget http://yourdomain.com/backup/backup.php -nc -q -O /dev/null" for an execution on sunday and wednesday at 2 am. Save and close the whole thing with ":wq" and that's it :-)

4 Nov
2008
Posted in: Eclipse, IDE, Infrastructure
By    No Comments

Eclipse PDT 2.0 Stable Release

Juhu - es ist da! Das erste stabile Release vom PDT Plugin für Eclipse Ganymede (na zumindest das erste, was ich da sehe ^^)

Laut Projektplan hätte gestern die M1 da sein sollen - vielleicht ist ja das Stable damit gemeint. Ich hoffe meine Vorfreude wird nicht getrübt, denn ausprobieren konnte ich es bisher noch nicht, das wird dann wohl morgen mal der Fall sein.

6 Aug
2008
Posted in: Eclipse, IDE, Infrastructure, PHP, Programming
By    2 Comments

Eclipse 3.4 Ganymede und PDT

  • english
  • german

Unfortunatly there is no working and official version of the PDT release for the current Eclipse release, Eclipse 3.4 Ganymede. However it is possible to use both. I found the solution within the Digital Base blog: Simply download the nightly build of PDT and integrate it as local site in the Eclipse update manager and then install. Quite easy.

16 Feb
2008
Posted in: Browser, Infrastructure
By    No Comments

IE6 auf PC mit IE7 testen

  • english
  • german

If you created webpages you probably want the page to look nearly the same in all browsers. Because there are still many Internet Explorer 6 users out there you should adjust your site for this browser, too. But that's not that easy if you have installed the IE7 because you don't have the IE6 anymore. But a VirtualPC image of microsoft finds a remedy. It comes with Windows XP and IE6 (or IE7 if you still have the IE6 and want to test IE7). Simply open it in the (free) VirtualPC software and enjoy the IE6 browserworld!

zum Download

5 Jan
2008
Posted in: Email, Infrastructure, PHP, Programming
By    250 Comments

Configure Mercury mail transport system for external mail

  • english
  • german

Usually I develope my projects locally on my pc. Needless to say that you sometimes have to send an email from your script to external mail to check if everything is fine. So you need to configure your system in a way that your local apache webserver is able to send mail to external addresses.
I always use XAMPP for Windows because it contains everything you need for developing PHP applications as well as all the stuff for sending mails: fake sendmail und Mercury Mail Transport System. I reference to XAMPP 1.6.5 (for Windows) and Mercury 4.5 here. Mercury is already pre-configured in XAMPP - all settings that I don't explicitly address stay as they are :)

I'm writing this because I'm not a mail professional and read a lot of tutorials and none of them really worked. So I extracted what I needed and put it all together. An important assumption for this tutorial is that you have your own SMTP server, for example the one that your webspace hoster provides.

So let's get started:

  1. start Mercury (using the XAMPP Controlpanel) and then open the admin panel.
  2. first of all we disable the HTTP server of Mercury so that it doesn't conflict with the apache:
    1. "Configuration" -> "Protocol modules"
    2. disable the check "MercuryB HTTP web server"
    3. I also disabled "Mercury IMAP4rev1 server" because I won't need that one
    4. leave the window opened, we'll need it immediatly
  3. to be able to send external mail we have to disable "MercuryE SMTP end-to-end delivery client" in the same dialog and enable "MercuryC SMTP relaying client". Click "OK" and restart Mercury!
  4. now let's configure Mercury in general:
    1. "Configuration" -> "Mercury core module", tab "General"
    2. we want to send from localhost, therefore we check if "localhost" is the value of "internet name for this system" and adjust that if not
    3. all other settings stay the same as they are already configured, but we'll have a look to the checkboxes beneath:
    4. disable all but "Send copies of all errors to postmaster"
    5. check under the tab "Local domains" if the entry "localhost (local host or server) localhost (internet name)" is there, if not, add it
    6. click "OK" and leave the dialog
  5. next we focus on the MercuryS SMTP Server for outgoing emails:
    1. "Configuration" -> "MercuryS SMTP Server"
    2. choose the tab "General" and add a wonderful name for your SMTP server under "Announce myself as", simply fill in any name, I've chosen "CS SMTP".
    3. under "Listen on TCP/IP port" fill in "25", that's the SMTP port
    4. add "127.0.0.1" to "IP interface to use", that's the local IP of your pc. With the IP of the pc within the network (192.168.0.X) it doesn't work (at least for me)
    5. now we limit the access to the server so that only our local machine can access it:
      1. under "Connection Control" click on "Add restriction" and add the IP range from "127.0.0.1" till "127.0.0.1" (it's that simple, isn't it?)...
      2. and select "Allow connections"...
      3. and leave all checkboxes deselected
    6. with a click on "OK" we quit the dialog and we're looking forward to the next one :)
  6. now let's configure the MercuryP POP3 Server:
    1. "Configuration"-> "MercuryP POP3 Server", select the tab "General"
    2. "Listen on TCP port" -> "110" and "IP interface to use" -> "127.0.0.1"
    3. choose the tab "Connection control" and proceed as already written under 5.5 (see above)
    4. that's it already, leave the dialog by clicking "OK"
  7. now we have the important one "MercuryC SMTP Client"
    1. "Configuration" -> "MercuryC SMTP Client"
    2. to send mail to external addresses we need to have an external SMTP server. If you're renting webspace somewhere and have mail included then you normally have access to a SMTP server. My webspace is hosted at HostEurope and I'll use the SMTP of my domain. Without such an external server the whole thing won't work because domain names won't be resolved among other things. I've chosen this path because I didn't feel like configuring around the whole day.
    3. enter the address of your SMTP under "Smart host name", for example "mail.meinedomain.com"
    4. depending on the way you access the server fill the values under "Connection port/type":
      1. for a "normal" SMTP that would probably be port 25 and "Normal (no SSL encryption)"
      2. I access my SMTP via SSL, that would be port 465 and "SSL encryption using direct connection"
      3. if you have other parameters simply try around a little :-)
    5. we finally fill in the "Login username" and the "Password" that normally is supplied by your webhost and we've nearly finished...
  8. let's check the Mercury users that are normally pre-configured:
    1. "Configuration"-> "Manage local users"
    2. there should be at least the users "Admin" and "postmaster", both with administrative rights. If not you have to add them.
  9. now we finished with Mercury, but we still need to configure PHP for sending mail with our scripts:
    1. we search and open the appropriate php.ini, using XAMPP you find it under "xampp/php/php.ini" in newer version and under  "xampp/apache/bin/php.ini" in older versions
    2. we search for "[mail function]"...
    3. and we add/adjust the following:
      1. "SMTP = localhost"
      2. "smtp_port = 25"
      3. "sendmail_from = postmaster@localhost"
    4. save the php.ini and restart the Apache
  10. now everything should work! But we'll test it first:
    1. within Mercury choose "File" -> "Send mail message" and send an email for testing purposes, I've chosen to send it to my googlemail account
    2. if the windows in Mercury are at sixes and sevens, choose "Window" -> "Tile" enjoy the view
    3. within the window "Mercury Core Prozess" we'll see our test mail at first:
      1. "13:38:41 JOB XXXXXX: from postmaster@localhost (local) To: XXX@googlemail.com (non-local) (JOB XXXXXX) -OK"
    4. the mail was received in Mercury and processed, after some seconds the window "Mercury SMTP client (relay version)" should show some actions:
      1. "05 Jan 2008 13:39, Servicing job XXXXXX ...OK"
      2. if you see this message, everything went fine and the mail was sent!
      3. if you don't get the message you have to find out why, possible reasons could be:
        1. wrong connection values for the SMTP server
        2. SMTP server doesn't allow relaying (from your host)
    5. now we'll test the whole thing from a PHP script and we'll write a wonderful one-liner into a PHP file:
      1. "mail('xxx@googlemail.com', 'Mercury test mail', 'If you can read this, everything was fine!');"
      2. call the PHP file within your browser, a command window should pop up shortly (or maybe not), it's from the fake sendmail of XAMPP, and focus back on Mercury:
        1. the produre is the same as above only that the SMTP server receives the mail from php before everything else happens
        2. you can watch this in the window "Mercury SMTP Server" and should see something like this: Mercury SMTP

I hope that worked for you, if not, feel free to leave a comment but as I already said, I'm not a professional but I'll help you as far as I can!

EDIT:
If you get the error message "SMTP server response: 553 We do not relay non-local mail, sorry." while sending from PHP go to Mercury under MercuryS -> Connection Control -> "Uncheck Do not Permit SMTP relaying to non-local mail" an check this option. Should fix the problem. Thanks to ron!